Data-Driven Approach for Spellchecking and Autocorrection

Abstract

This article presents an approach for spellchecking and autocorrection using web data for morphologically complex languages (in the case of Kazakh language), which can be considered an end-to-end approach that does not require any manually annotated word–error pairs. A sizable web of noisy data is crawled and used as a base to infer the knowledge of misspellings with their correct forms. Using the extracted corpus, a sub-string error model with a context model for morphologically complex languages are trained separately, then these two models are integrated with a regularization parameter. A sub-string alignment model is applied to extract symmetric and non-symmetric patterns in two sequences of word–error pairs. The model calculates the probability for symmetric and non-symmetric patterns of a given misspelling and its candidates to obtain a suggestion list. Based on the proposed method, a Kazakh Spellchecking and Autocorrection system is developed, which we refer to as QazSpell. Several experiments are conducted to evaluate the proposed approach from different angles. The results show that the proposed approach achieves a good outcome when only using the error model, and the performance is boosted after integrating the context model. In addition, the developed system, QazSpell, outperforms the commercial analogs in terms of overall accuracy.

1. Introduction

Spellchecking is a task of automatically finding misspelled words in a document; the results of a spellchecking system could be presented to the user as underlining misspelled words. Autocorrection is a task of suggesting and substituting corresponding well-spelled words forms for misspellings. A spellchecking and autocorrection system is widely applicable for many fields, such as news article and book editing, message text and official document writing, etc. It is also applicable for prepossessing textual documents [1] and post-processing optical character recognition (OCR) [2].

For developing a spellchecking and auto-correction system for any language, it is essential to have an annotated misspellings corpus, where the information on misspelled words and their well-spelled forms are presented. Constructing this type of manually annotated corpora is time consuming and often expensive. It is hard to guarantee that the annotated corpus is balanced and covers most of the typical word errors. Kazakh is a morphologically complex language in which a single root produces hundreds and thousand of new words by attaching suffixes to it. Due to these characteristics of the language, it is impossible to store whole Kazakh word variations in a dictionary, and it covers many misspellings by manually annotating a corpus.

For instance, Figure 1 shows the variations of real misspellings which are found in the Kazakh web news automatically. It can be seen that there are many-to-one relations, which means that manually finding all those actual misspellings is time consuming and expensive. For morphologically complex language (MCL), such as Kazakh, Turkish, Finnish, etc., because of their rich variability in word forms, it is hard to manually annotate a spelling correction corpus.

Existing Kazakh spelling correction systems utilize an annotated corpus, and most of these systems utilize replacement rules [3], Levenshtein distances [4], and morphological analyzers plus noisy channels [5], which is summarized in

Table 1. Existing work for Kazakh spelling correction. FSA—finite state automata.

Studies	Method	Accuracy
Makazhanov et al. [5]	FSA + noisy channel	37%, 1-best 55%, 2-best 67%, 3-best
Slamova and Mukhanova [3]	Rules + Levenshtein	85.4%
Abdussaitova and Amangeldiyeva [4]	(i) Levenshtein-based; (ii) Levenshtein + classification; (iii) Naive-Bayes	(i) 62% (ii) 71.3% (iii) 89.38%

. To our knowledge, the existing systems for Kazakh language are not publicly available along with the annotated corpus. One of the existing systems’ limitations is that they are based on the fully string model, not a sub-string one. Facing the large variability of Kazakh words, a fully string model suffers from the high-data sparseness problem caused by the language agglutinative nature.

In this paper, we present a data-driven, sub-string approach for Kazakh spellchecking and autocorrection, which do not require any manually annotated corpora and any explicitly compiled dictionary of well-spelled words. Kazakh is a less-resourced language NLP-wise. The main purpose of this work is as follows: (i) to develop a sub-string spelling correction system for Kazakh, which does not require any explicitly annotated corpus and can be applied to other MCLs; (ii) to create a Kazakh word-error pairs corpus from a large noisy corpus for future use; and (iii) to produce a strong baseline work for Kazakh spelling correction, which is easy to follow.

Recent general studies on spelling correction are based on more complex neural networks, for example, Gan et al. [6] presented a spelling error correction model with a soft-masked BERT model [7] in a curriculum learning (CL) manner. Ji et al. [8] proposed a SpellBERT which is four layers of Bert, but only the half size of vanilla BERT, and can show competitive performance and produce a state-of-the-art result on the OCR dataset. The proposed competitive spelling correction approach for heavily resourced languages (such as English, Chinese, etc.) is an excellent scientific contribution. However, considering the purpose of this article and making the work fit to less being more principal, for a less-resourced language, such as Kazakh, a spelling correction model along with its dataset is more necessary.

An error model [9] is a simple and sub-string approach that has achieved good outcomes for many languages. We apply this sub-string method to Kazakh spelling correction task as a base model in this work. Then a context model of Kazakh text is integrated into the sub-string model to improve the system’s performance further. This approach requires an annotated Kazakh corpus of word–error pairs. To solve this problem, we propose a purely data-driven approach that infers the knowledge of misspelling with its correct fix from sizable noisy web data.

The proposed approach consists of five stages:

• Term filtering—instead of building a well-spelled lexicon, a term list is collected in a simple filtering manner.
• Triple inferring stage—the useful information about misspellings terms is inferred from observing the noisy web data.
• Alignment of the misspelling and the intended words; this is used to build an error model.
• To enhance the error model, we build an n-gram language model for Kazakh, which makes the model with context-appropriate corrections.
• Candidate scoring step—develop and optimize the mixture of the error model and language model; as a result, our system can detect and correct real-word substitutions, i.e., word usage and grammatical errors.

To inter triples, we introduce a context-based filtering approach that first filters terms list with frequency using the Levenshtein–Damerau algorithm to infer orthographically similar terms then uses a context-shared strategy to infer those misspellings. For a MCL (such as Kazakh), inferring misspelled words from noisy web data is a challenging task due to the agglutinative nature of language. A window-based error model is introduced, which makes an alignment between misspellings and intended words at the sub-string level, and calculates the probabilities of how likely it is that sub-strings from misspelled words transfer to sub-strings of intended words. Several experiments are conducted to evaluate the proposed approach, and the results show that QazSpell outperforms the baseline that is a commercial analog.

The rest of the paper is organized as follows: Section 1 introduces the work, and Section 2 describes the related work. Section 3 introduces the proposed system, followed by the experiments and results reported in Section 4. Finally, we conclude the work with future work in Section 5.

2. Related Work

This section provides an overview of prior work on the correction of spelling errors in general. A spelling correction system consists of three stages of processing:

• Detects a spelling error;
• Generating a set of candidates for correction;
• Ranking the candidates.

Below, we describe related early work regarding these three angles. Then, we introduce the existing studies for the Kazakh spelling correction task to compare it with this work.

2.1. Error Detection

Spellchecking is one of the typical problems in natural language processing, and one of the early works of [10] provided many solutions; most approaches are based on the use of one or more manually compiled resources.

A word is either a common word or a lower-frequency proper name, or it may be borrowed from another language. According a misspelled word belongs to a dictionary of correct words or not, word errors can be divided into two classes [10]:

• Real-word errors, where a word is misspelled, but its misspelled form is correct in the language.
• Non-word errors, where a word is misspelled and its form is not in the dictionary.

A fast lookup technique can be applied for non-word error detection, such as hash table [11] or search tree [12,13], and a misspelled word can be searched for in a dictionary. The most commonly used open-source spelling correction systems are Aspell (http://aspell.net, accessed on 29 July 2022) and Hunspell (http://hunspell.github.io/, accessed on 29 July 2022), which are used for detecting non-word errors in most of cases. Real-word errors are hard to detect than non-word errors because it requires semantic analysis of context. Mashod Rana et al. [14] proposed an autocorrection system for the Bangla language using a language model to correct a homophonic real-word error. Deorowicz and Ciura [15] claimed that creating a lexicon for all correct words could be too large and it can lead to misdetection and many real-word errors. Wang and Liao [16] projected each word in a test sentence into a high dimensional vector space in order to reveal and examine their relationships by using a conditional random field (CRF)-based detector.

Levenshtein–Damerau edit distance was introduced in the work [17] as a standard way to detect spelling errors. The noisy channel model was applied to the spelling correction task by Kernighan et al. [18] and Church and Gale [19]. The noisy channel model is a kind of Bayesian inference, as it calculates the probability of seeing a misspelled word and trains the parameters with the condition of how a word generates this misspelled word. A sub-string error model [9] was used for English spellchecking and autocorrection, which proposed to use a context window to partition misspelled and correct words, and using the partitions to calculate the probability of a partition of a source word conditioned to a partition of its candidate. It also adds a positional feature to the condition, which shows better results. Besides using the edit distance as features, solving the phonetic errors is another challenging task. Phonetic spelling errors could lead to a large edit distance with variously noisy forms. Atkinson [20] attempted to address phonetic errors by generating sound-like equivalent candidates based on phonetic algorithms. In this direction, Yang et al. [21] presented a generalized spelling correction system integrating phonetics to address phonetic errors in E-commerce search without additional latency cost. Zhang et al. [22] presented an end-to-end Chinese spelling correction model that integrates phonetic features into a language model by leveraging the powerful pre-training and fine-tuning method. Additionally, neural network-based spelling correction approaches [8,23] were proposed using sophisticated neural networks.

2.2. Candidate Generation

All words in a lexicon of correct words can be used as candidates after the detection process. To further optimize this process, it is reasonable to restrict the search space that are similar to the detected misspellings. Zhang and Zhang [24] proposed EmbedJoin, an approach to the problem of edit similarity joining, which calculates the edit similarity for two strings given a threshold value. Kinaci [25] used a character-level language model trained on a dictionary of correct words to generate a candidate list. Reffle [26] used a Levenshtein automata to efficiently determine word similarity. Yu et al. [27] provided a comprehensive survey on the methods of string similarity search and join.

2.3. Candidate Ranking

A noisy channel model [28] calculates similarity between two strings as a probability of converting one string into another. Brill and Moore [9] applied a noisy channel to produce a list of possible correction candidates and rank them by the obtained probability score. Except the ranking candidate, only considering the comparing of two strings, the context information can be applied in the ranking, which refers to the context model. It takes into account the misspelled word’s context and makes the correct words appear on the top position in a list of candidates, which has semantic relevance to the identified misspelled word. Flor [29] presented an research work of using four types of contextual information for improving the accuracy of autocorrection system: (i) immediate local context; (ii) local lexical neighborhood using a very large distributional semantic model; (iii) recognizing a misspelling as an instance of a recurring word can be useful for re-ranking; and (vi) looks at context beyond the text itself.

2.4. Spelling Correction for Kazakh

For Kazakh spelling correction, the author [5] proposed a spelling correction tool for Kazakh language based on a morphological disambiguator [30]. Makazhanov et al. [5] reported the proposed method outperformed both open-source and commercial analogues, achieving the overall accuracy of 83% in generating correct suggestions. This approach requires a manually created lexicon, knowledge about the language, and an annotated training corpus. They collected more than 1800 error–correction pairs from the annotated Kazakh language corpus (KLC) [31]. The proposed method produces 37%, 55% and 67% accuracies for 1-$best$, 2-$best$ and 3-$best$ suggestions.

Slamova and Mukhanova [3] presented a text normalization and spelling correction method for Kazakh language by applying a set of replacement rules as regular expression patterns and keyboard errors with the Damerau distance calculation. For the 500-word test set, their correction system archived 85.4% accuracy. Abdussaitova and Amangeldiyeva [4] proposed a Kazakh text normalization study using three methods: (i) the Levenshtein-based algorithm, (ii) Levenshtein plus classification rules-based algorithm and (iii) naive-Bayes-based algorithm. The dataset was collected from a survey and Kazakh websites. The approximate size of the dataset was about 110 thousand words, and it includes the most common, ill-formed, and spoken words. The accuracies of their three methods are 62%, 71.3%, and 89.38%, respectively. Table 1 summarizes the existing studies for Kazakh spelling correction for reference. Most Kazakh spelling correction approaches require a compiled lexicon containing well-spelled words and manually annotated error–correct word pairs as the training corpus. Existing Kazakh spelling correction systems are rule-based or distance metric based, and the naive Bayes one used a fully string model, not a sub-string one. None use a context model, which plays a vital role in correcting contextual word errors in practice. The corpus collected by the previous work is not publicly available. The proposed approach in this article has several differences: (i) It does not require any manually annotated corpora and any explicitly compiled dictionary of well-spelled words. The training data are extracted from the noisy web data without the use of any manually annotated corpus. (ii) The proposed method is a purely data-driven approach that does not require any replacement rule or manually annotating the data. (iii) A sub-string model integrated with a context model is presented. Due to the agglutinative nature of the Kazakh language, using a sub-string model is reasonable, rather than using a full-string model.

3. Methodology

This section describes our methodology (with an example of Kazakh) for spellchecking and autocorrection using noisy web data without the use of any manually annotated corpus. The proposed approach consists of five steps described in the following subsections. Figure 2 shows the process of spellchecking and autocorrection.

3.1. Term List

To score the candidate list for correction, we require a list of well-spelled words. Rather than attempt to build a dictionary that contains well-spelled words; instead, we take the most frequent term observed on the web. The idea behind this attempt is that the correctly spelled words mostly appeared compared to the misspelled ones. To do this, we crawl a large Kazakh news article from the web, and then we apply several steps of pre-processing, namely by tokenization [32], removing punctuation, numbers, and special characters. We created a list of candidate words with simple frequency-based filtering by taking the most frequent term observed on the web. We do not set a large frequency threshold for this due to the agglutinative nature of the language (hundreds and thousands of new words can be produced from a single root), and the word frequency for Kazakh has a long tail effect, which means most of the words’ frequencies are small and similar to each other. The obtained term list is so large (∼2 million tokens), and it should contain well-spelled words but also contain misspelled words.

3.2. Using Web to Infer Misspellings

To build an error model, we require a training dataset with a set of triples (misspelled word, intended word, and count), which is also derived from the large web data. We believe the web data are ideal for inferring the triples because most of the web data are generated by users, and they contain well-spelled words and real misspelled words. In the autocorrection process, we do not directly use indented words in triples cause our method models sub-strings from well-spelled and misspelled words. To find the triples, we make two assumptions: (i) misspelled words are orthographically similar to indeed words, which is found by [17], who observed that 80% of 892 misspellings is derived from single instances of insertion, deletion, or substitution. (ii) Most words are usually well spelled on the web and can be treated as intended words.

To extract the triples, at first, we use the obtained term list. For each word in the term list, we find all other terms in the list that have small orthographic similarities. We define the similarity using the Levenshtein–Damerau edit distance, which is used with restriction in the following manner: for word length, ≤4, the edit distance is set to one, and if the word length ranges within $4\sim 12$, the edit distance is set to two and set to three for longer words. To speed up the computation of the edit distances process, we compile the term list into a tree-based data structure that allows for efficient searching on the entire term list with a maximum edit distance. We also use multiprocessing to parallel the computations to further speed up the distance calculation.

At this stage, for each term in the list, we find a cluster of terms that are orthographically similar to it. One of the issues in the obtained misspellings is that they still contain many well-spelled words that only differ from the intended word with added suffixes. To tackle this issue, a context-shared filtering algorithm is introduced, which is based on the following two assumptions:

• Most of the time, intended words occur more in context than misspellings.
• Misspellings are fewer in occurrence in the context and have a similar context with the intended word.

We use the above assumptions to identify misspelled and intended words by filtering the triples obtained from the edit distance filtering stage. Algorithm 1 shows how to filter misspelled and intended words.

Algorithm 1: Filtering misspelled and intended terms.

Input:  a term list $T=(w^{\prime },\left\{w\right\})$, a set of contexts C and a threshold $th$.  Output:  $D=(w,\{s\left\}\right)$.   1: for <($w^{\prime },\left\{w\right\})\in T$> do   2:     $\left\{ms\right\}\leftarrow \varnothing$   3:     for <$t_i\in \left\{w\right\}$> do   4:         $count\left[i\right]\leftarrow C\left[t_i\right]$   5:     end for   6:     $iw\leftarrow \left\{w\right\}[arg{max}_{\left(count\right)}]$ using first assumption to find the intended word.   7:     for <$t_i\in \left\{w\right\}$> do   8:         if $t_i\ne iw$ then   9:            $sharedcontext\leftarrow len(C\left[t_i\right]\cup C\left[iw\right])$ using second assumption to find the misspellings.   10:            if $sharedcontext\ge th$ then   11:                $\left\{ms\right\}\leftarrow t_i$   12:            end if   13:         end if   14:     end for   15:     $D\leftarrow (iw,\{ms\left\}\right)$   16: end for

3.3. Error Model

Our spellchecking model is based on a noisy channel model of spelling errors [19]. For an observed word w, the model should generate a list of candidate words s by the following calculations:

$$P\left(s\right|w)=P(w\left|s\right)\times P\left(s\right)$$

(1)

where the Bayes’ rule was applied and the constant denominator was dropped. We obtain the unnormalized posterior as a noisy channel model with two components: a source model $P\left(s\right)$ and a channel model $P\left(w\right|s)$.

Instead of using noisy channel model on entire words, we use a sub-string error model to calculate the probability of $P\left(w\right|s)$. To derive such a sub-string error model, we define R to be all possible adjacent partitionings of s and similarly let T be the partitioning of w, where $\left|R\right|=\left|T\right|$. The partitioning process is in one-to-one alignment and also allows the partition to be empty. For particular partitions R and T, the error model [9] estimates $P\left(w\right|s)$ as follows:

$$P\left(w\right|s)=ma{x}_{R,Ts.t.\left|R\right|=\left|T\right|}\prod _i^{\left|T\right|}P\left(T_i\right|R_i)$$

(2)

We train the error model by using the triples (misspelled word, intended word, and count) derived in the above steps and use the maximum likelihood estimates of $P\left(T_i\right|R_i)$. Figure 3 shows an example of alignment with one edit distance. It can be seen that there are symmetric and non-symmetry patterns that can be obtained from two sequences, and the probability for those patterns should be calculated by an error model.

Two string are aligned by minimizing the edit distances based on single character insertions, deletions and substitutions. It can be seen from Figure 3, for one edit distance alignment, it corresponds to the sequence of the following symmetric and non-symmetric patterns:

$$c\to c,o\to o,r\to r,o\to o,n\to k,\epsilon \to a,a\to a,v\to v,i\to i,r\to r,u\to u,s\to s$$

To obtain richer context information, the window size can be increased, then for the first mismatch of the above example, the model can generate the following pairs (in case of edit distance = 3):

$$n\to k,on\to ok,n\to ka,on\to oka,na\to kaa$$

For nonmatch edits, we will get similar alignments, then we can calculate the probability of them. It indicates how likely a right-side sub-string can be produced by the left-side sub-string. In practice, it can be done by counting those two sub-strings from the corpus for estimation.

In this way, we can obtain a error model which calculates the probability for misspelling words and its candidates: how likely it is that the candidate fits to the misspelled word with the sub-string model. Currently, the error model can generate candidates by calculating sub-string probabilities that only capture the word orthographic features.

3.4. Context Model

In order to detect real-word errors and to improve the error model, only capturing orthographic features is not enough; the semantic analysis of the context is required to detect and generate candidate words. Another benefit of using a language model is reducing the searching space, as there is no need to calculate the similarity score for all lexicons derived from the web.

For this purpose, we incorporate a language model to the error model. We use the n-gram language model with Stupid Back-off [33] to estimate the second part $p\left(s\right)$ of Equation (1). We use a large text corpus to train a trigram language model which can be formulated as follows:

$$P(w_1,...,w_m)=\prod _i^{m}P\left(w_i\right|w_1,...,w_{i-1})$$

(3)

3.5. Ranking

How to rank the obtained candidates is an important issue in an autocorrection system, which affects the performance of the final results significantly. We score each candidate word by combining the error model’s score and the language model’s score.

The final score for each candidate correction is computed as follows:

$$P\left(s\right|w)=\lambda ·\stackrel{\mathrm{error}\mathrm{model}}{\overbrace{P\left(w\right|s)}}\times (1-\lambda )·\stackrel{\mathrm{context}\mathrm{model}}{\overbrace{P\left(s\right)}}$$

(4)

Hyper-parameter $\lambda$ is tuned during the experiments.

4. Experiment

To evaluate the QazSpell performance, we conducted a set of experiments: (i) hyper-parameter tuning experiment; (ii) investigation of the accuracy of QazSpell for x-best candidates and n-window (edit distance); (iii) a comparison experiment of QazSpell using a language model and not using it; (iv) a comparison experiment with a commercial analogue, where we compare QazSpell with the Microsoft office for Mac (MSO) version 16.56 as the baseline; and (v) an evaluation of the stability of QazSpell by increasing the test data.

4.1. Data Set

We crawled a collection of Kazakh news text from different sources, shown in

Table 2. Crawled web data sources used in QazSpell (accessed on 3 May 2020).

’http://today.kz’, ’http://vesti.kz/’, ’http://www.kp.kz/’, ’http://www.newsfactory.kz/’, ’http://www.spik.kz/’, ’https://24.kz/ru/’, ’https://365info.kz/’, ’https://aif-kaz.kz/’, ’https://bnews.kz/ru(baigenews.kz)’, ’https://forbes.kz/’, ’https://inbusiness.kz/ru’, ’https://kapital.kz/’, ’https://kazakh-tv.kz/ru’, ’https://kaztrk.kz/ru’, ’https://liter.kz/’, ’https://rus.azattyq.org/’, ’https://sputniknews.kz/’, ’https://tengrinews.kz/’, ’https://www.kt.kz/’, ’https://www.ktk.kz/’, ’https://www.nur.kz/’, ’https://www.zakon.kz/’

. The obtained web data were pre-processed with several steps: (i) removing numbers, punctuation, emails, links and extra special characters; and (ii) applying sentence and word segmentation. The statistics of the obtained data are shown in

Table 3. Data statistics.

documents	244,277
tokens	2,043,209
triples	27,228

. It can be seen that there are over 2 million tokens as the term list, and 27,228 error–correction pairs are filtered out after applying Algorithm 1.

To evaluate the models, we randomly selected 500 words from the corpus with its left context in order to compare the system’s performance under the condition of adding a language model. For each selected word, we made a set of synthetic misspelled words by randomly taking one of the following operations:

• Inserting a random character in a random position of the word;
• Substituting a random character of the word with a random character;
• Deleting a random character from the word;
• Exchanging the position of a random character with another random character.

In total, 500 error–correction pairs with their context were obtained as the test for evaluation. QazSpell was compared with Microsoft Office manually; we put 500 word error with their trigram context into Microsoft Word and manually checked if it correctly identified and corrected the misspellings.

4.2. Results

We begin with reporting the hyper-parameter tuning process; a small set of test data was chosen for this experiment. To tune a parameter $\lambda \in [0,1]$, we heuristically choose a set of its values: [0.1, 0.12, 0.13, 0.15, 0.2, 0.3, 0.4, 0.7]. Figure 4 plots how the accuracy of QazSpell is performed when setting the various value of $\lambda$. The result shows that the model performs best at the value of 0.15. The model’s accuracy decreases when taking larger or smaller values than that. Thus, in the following experiments, we set hyper-parameter $\lambda$ to 0.15.

Second, we analyze QazSpell by important metrics: x-$best$ and n-window. The former is calculated as a percentage of the correct suggestion that appeared at the first x-positions of the ranked suggestion lists. The latter is calculated when the model is trained for different maximum context window sizes.

Table 4. Autocorrection accuracy without a language model.

x-best\Window	1	2	3	4	5
1	0.664	0.636	0.654	0.654	0.656
2	0.77	0.76	0.77	0.77	0.77
3	0.824	0.816	0.83	0.83	0.83
4	0.862	0.86	0.868	0.868	0.868
5	0.882	0.884	0.888	0.89	0.89
6	0.89	0.9	0.902	0.904	0.904
7	0.906	0.912	0.914	0.914	0.914
8	0.914	0.924	0.92	0.92	0.92
9	0.924	0.93	0.93	0.93	0.93
10	0.93	0.934	0.936	0.936	0.936

shows the accuracy results of QazSpell without a language model. It can be seen that for the x-best evaluation, the model produces ≈ 0.65 accuracy for 1-$best$. The model gains a large improvement in 2-$best$ (≈11%), and 3-$best$ (≈5%). For rest of them, QazSpell gains minor improvements, and at 10-$best$, it reaches a high result of accuracy ≈93.

Table 5. Autocorrection accuracy with a language model.

x-best\Window	1	2	3	4	5
1	0.796	0.786	0.786	0.786	0.788
2	0.914	0.91	0.912	0.912	0.912
3	0.938	0.936	0.94	0.942	0.942
4	0.944	0.95	0.952	0.952	0.952
5	0.948	0.952	0.952	0.952	0.952
6	0.95	0.952	0.952	0.952	0.952
7	0.954	0.954	0.954	0.954	0.954
8	0.954	0.954	0.954	0.954	0.954
9	0.958	0.958	0.958	0.958	0.958
10	0.96	0.96	0.96	0.96	0.96

shows the accuracy results of QazSpell with a language model. As we can see, at 2-$best$, QazSpell is significantly improved (≈12) compared to 1-$best$. For the rest of x-$best$, we can observe slight improvements over its previous ones. The results are different from QazSpell without a language model, and the former one has two significant gains in 2-$best$ and 3-$best$, which may indicate that a language model supplements the error model. In these experiments, we did not observe a huge difference when using various value for the window.

Third, we compare QazSpell’s performance with and without a language model. Figure 5 plots the accuracy values of QazSpell when using a language model and not using it. It can be seen that the QazSpell’s performance is sharpened after using a language model, which gains $\approx 15\%$ improvement at 1-$best$ compared to the model not enhanced with a language model. With the growth of x-$best$, the accuracies tend to be close to each other, but we still observe the advantages of using a language model.

Next, we calculate the overall accuracy for both systems, and the accuracy is calculated by ignoring the position of the correct fix in the suggestion list. The value can be considered the upper bound of QazSpell. QazSpell outperforms MSO in terms of overall accuracy; the former obtains 0.974%, and the latter obtains 0.694%, respectively.

Let us move on to compare both accuracies on x-$best$ since several of the top candidates are often important in the practical use of an autocorrection system. Figure 6 shows the accuracy results of QazSpell and the baseline of MSO. Since MSO suggests three candidates at maximum, we report $[1,2,3]$-$best$ for the comparisons. It can be seen that QazSpell outperforms MSO, whether it uses a language model or not. MSO obtains 0.584% accuracy for 1-$best$, 0.658% and 0.694% for $[2,3]$-$best$. QazSpell obtains $\approx 20\%\sim 25\%$ improvement over MSO.

In the above experiments, we tested the proposed approach with a small error-correction corpus of about 500 words. In order to further evaluate the stability of QazSpell, we reassembled the test data with the same procedure described in Section 4.1; in this case, the size of the dataset is increased gradually. We collected a different-sized error-correction corpus: $D=[1000,2000,3000,4000,5000]$ with unique words, and ran the trained QazSpell to correct the misspellings. The accuracy results of QazSpell for using a language model or not using it are reported separately. This purpose is to individually evaluate the proposed error model’s stability.

Table 6. Evaluation of the stability of QazSpell without a language model.

x-best\Corpus Size	1k	2k	3k	4k	5k
1	0.647	0.672	0.693	0.690	0.697
2	0.771	0.788	0.805	0.805	0.816
3	0.827	0.841	0.851	0.852	0.859
4	0.859	0.874	0.881	0.882	0.889
5	0.888	0.898	0.903	0.903	0.909
6	0.899	0.908	0.913	0.913	0.919
7	0.911	0.921	0.924	0.925	0.930
8	0.924	0.93	0.933	0.933	0.938
9	0.936	0.942	0.943	0.941	0.945
10	0.944	0.947	0.947	0.945	0.949

shows the accuracy results of QazSpell for different sized test sets without using a language model. It can be seen that for 1-$best$, the accuracy of QazSpell is varied from 0.647% (1000 words) to 0.697% (5000 words). The accuracy is not decreased when testing more data, which may indicate the robustness of the proposed error model. For other x-$best$, minor changes appear in the accuracy when the test data size is increased.

Table 7. Evaluation of the stability of QazSpell with a language model.

x-best\ Corpus Size	1k	2k	3k	4k	5k
1	0.786	0.791	0.791	0.782	0.783
2	0.901	0.909	0.907	0.903	0.906
3	0.929	0.936	0.930	0.929	0.932
4	0.942	0.948	0.943	0.943	0.945
5	0.947	0.951	0.949	0.949	0.951
6	0.955	0.958	0.954	0.953	0.955
7	0.956	0.959	0.956	0.954	0.956
8	0.958	0.960	0.958	0.956	0.958
9	0.963	0.964	0.961	0.959	0.961
10	0.965	0.966	0.963	0.960	0.962

reports the accuracy results of QazSpell with different sized test sets when adding a language model. It can be seen that with the growth of the test set, the variety of the accuracy is minimal. For the 1-$best$ candidate, QazSpell achieves around 0.78%∼0.79% accuracy over the different sized test set from 1000 to 5000 words. In the 2-$best$ position, the proposed approach achieves a significant improvement over the 1-$best$ for all different sized-test sets. After 2-$best$, there are minor improvements that can be observed. Overall, whether the QazSpell uses a language model or not, the results indicate that the proposed approach has good stability performance when a large test set is applied.

The speed of QazSpell was evaluated on a computer of the following environment: MacBook Pro, Apple M1 Pro, 16GB. We found that a direct aligning sub-string model works very slowly in practice, especially when there are sizable candidates that should be ranked. Instead of aligning the sequences directly, QazSpell’s speed was optimized with a tree-structure-based sub-string model. With this optimization, QazSpell took 73 CPU seconds to correct the 1000 misspellings. No parallelism techniques were applied in this evaluation.

5. Conclusions

Most of the existing spelling correction systems for Kazakh are developed on an annotated word–error pair corpus, and their systems with their annotated data are not publicly available. Fully string approaches are applied in the existing Kazakh spelling correction systems, which suffer from the data sparseness problem caused by the agglutinative nature of language. Kazakh is a morphologically complex language, in which, a single root may produce hundreds and thousands of new words, considering that the Kazakh language is a less-resourced language, NLP-wise.In this article, we represent a data-driven, sub-string approach for spellchecking and autocorrection for morphologically complex languages (MCL) in the case of the Kazakh language.

The main feature of the proposed approach is that it does not require any manually annotated corpora nor any explicitly compiled dictionary of well-spelled words. Second, unlike the existing Kazakh spelling correction system that applies a fully string model based on replacement rules, Levenshtein distances and noisy channel, the proposed approach is a sub-string error model integrated with a context model, which can be a new baseline for the Kazakh spelling correction task. Third, a corpus of Kazakh word–error pairs from a large noisy corpus is created for future use.

The triples of <misspelled words, intended words, count> from a sizable noisy web data are inferred and used to train a sub-string error model. A context model is trained on a large corpus and integrated with the sub-string model to further improve its performance. A set of experiments is conducted to evaluate the proposed approach from different angles: (i) tuning the regularization of the error model and the context model by a hyper-parameter; (ii) evaluating the proposed model of a sub-string model and sub-string +context model in terms of accuracy at x-best, independently; (iii) comparing the proposed model with a commercial analogy; (iv) evaluating the stability of the proposed approach by increasing the size of test set; (v) testing the running speed of QazSpell. Experimental results show that the sub-string model achieves comparable results when comparing with the existing Kazakh spelling correction system (because there is no shared test set, this is only for reference). The sub-string error model gains a significant improvement when using a context model, and it outperforms a commercial analogue. The stability evaluation result shows that QazSpell results are stable when increasing the test set size, whether it uses a context model or not. Without using any parallelism techniques, only using one CPU core, QazSpell took 73 CPU seconds to process the 1000 misspelled words.

Although these results satisfy the study’s purpose, we found several limitations of QazSpell: (i) Because the word–error pairs derived from a sizable noisy web data, there are false misspellings that exist in the corpus (this is reasonable due the agglutinative nature of languages), which bring noisy information to the model. (ii) The extracted triples and the tokens are still in the surface word forms, and the current sub-string model may not fully overcome the sparseness problem caused by language. One possible solution is using a heuristic method to detect the lemma border to a misspelling then apply two separate sub-string models to the lemma and suffixes part, then integrate to a global error model.

There are several lines of possible future work: (i) propose a cascade sub-string error model to overcome the second limitation of QazSpell mentioned above; (ii) propose a neural network-based (several architectures will be applied, such as CNN [34,35], long short-term memory [36], Bert [7], etc.) error model to further improve the performance; (iii) apply different methods to identify more sophisticated errors, such as grammar, cognitive spelling errors, etc.