A Mandarin Tone Recognition Algorithm Based on Random Forest and Feature Fusion ^† †

Abstract

In human–computer interaction (HCI) systems for Mandarin learning, tone recognition is of great importance. A brand-new tone recognition method based on random forest (RF) and feature fusion is proposed in this study. Firstly, three fusion feature sets (FFSs) were created by using different fusion methods on sound source features linked to Mandarin syllable tone. Following the construction of the CART decision trees using the three FFSs, modeling and optimization of the corresponding RF tone classifiers were performed. The method was tested and evaluated on the Syllable Corpus of Standard Chinese (SCSC), which is a speaker-independent Mandarin monosyllable corpus. Additionally, the effects were also assessed on small sample sets. The results show that the tone recognition algorithm can achieve high tone recognition accuracy and has good generalization capability and classification ability with unbalanced data. This indicates that the proposed approach is highly efficient and robust and is appropriate for mobile HCI learning systems.

1. Introduction

Different from English or other Western non-tonal languages, monosyllables with the same pinyin in Mandarin have four tones, such as mother (ma1), hemp (ma2), horse (ma3), and abuse (ma4). Therefore, for a learner whose mother tongue is non-tonal, mastering the tonal pronunciation of monosyllables is both a difficult and key point in learning Mandarin [1]; even a Chinese child with severe and profound prelinguistic deafness, after the implantation of a cochlear implant (CI) device, would also have a similar problem [2]. Since Mandarin sentences are composed of many continuous monosyllabic words and due to the vocal cord vibration inertia and mutual influence during the pronunciation of adjacent syllables, some syllables’ tones will change based on the tone-changed rules. A portable Mandarin tone training system with human–computer interaction (HCI) function is of great significance for learners. However, the tone recognition algorithm should be low complexity, efficient, robust, and speaker independent.

The key to tone recognition lies in the extraction of feature parameters and the design of the tone classifier. There are already a number of tone recognition approaches which are either based on machine learning or on deep learning. The method proposed by Fu et al. [3] extracted features related to fundamental frequency and energy in voiced segments and used support vector machines (SVMs) to automatically recognize four tones in Mandarin, achieving 93.52% accuracy. In Mizo tone recognition with four tones as in Mandarin and with six F0 features, the SVM-based classifier achieved 73.39% accuracy and the deep neural network (DNN)-based classifier achieved 74.11% accuracy [4]. However, the extracted sound source features in the above two methods were not fully optimized and fused. Zheng et al. [5] proposed a tone recognition algorithm for Mandarin three-syllable words based on fundamental frequency and an improved back propagation neural network (BPNN) algorithm, and the accuracy of the first word, middle word, and last word was 87.50%, 70.83%, and 79.17%, respectively, though the BPNN model was obtained by training involving 800,000 epochs. Shen et al. [6] put forward a Chinese four-tone recognition method based on the fusion of prosodic and cepstral features, and the tone classification accuracies of the Gaussian mixture model, BPNN, SVM, and convolutional neural network (CNN) were 84.55%, 86.28%, 85.50%, and 87.60%, respectively. Liu et al. [7] proposed a one-step continuous Mandarin tone recognition method, and the final tone accuracy rate was 88.80% based on pitch features and spectral features by using an MSD-HMM (multi-space distribution hidden Markov model). The performance of these two algorithms is still not ideal and the construction of tone feature parameters is not optimized for tone recognition tasks. In [8], a speaker-independent four-tone recognition system for Mandarin digits was realized that was only based on the pitch contour of each syllable, which achieved 90.20% recognition accuracy and a response time of about 1.64 s. Although the system is very simple, the accuracy and response time still need further improvement. A CNN architecture Mandarin tone classifier [9] was based on the features from pre-training on Mel frequency cepstrum coefficient (MFCC) vectors through the use of a denoising autoencoder and achieved 95.53% accuracy; however, the use of 3600 syllables was required for training. Based on a CNN and multi-layer perceptron, the ToneNet model [10] was used to classify Mandarin monosyllabic tones and achieved 99.16% accuracy with the Syllable Corpus of Standard Chinese (SCSC). However, the method used the Mel spectrogram’s image as the input of the model and required a large amount of data for training and large computation.

Although some of the above methods have achieved good results in Mandarin tone recognition, there is still no method that combines high accuracy and low complexity. While the random forest (RF) is an efficient ensemble modeling approach, in the process of model construction, the bootstrap aggregating (bagging) algorithm and random feature selection strategy can be used to avoid falling into local optimization [11,12]. There is also no Mandarin tone recognition method based on RF and feature fusion, and how to construct appropriate fusion features for a RF classifier and its recognition performance remains to be explored. Thus, in this paper, we introduce RF into tone recognition to establish a highly efficient Mandarin tone recognition method with a high level of performance, which should be suitable for mobile training systems, especially for small sample sets.

In the study, the sound source features related to Mandarin tones are first comprehensively selected and fused, and the fusion feature sets (FFSs) are produced by using three fusion methods, which are respectively used to form corresponding RF models, with the classification and comparison experiments then conducted. Furthermore, the tone recognition performances of different FFSs and RF with small sample sets are evaluated. The results show that the RF classifier has high effectiveness and robustness, which is suitable for Mandarin tone recognition in small sample sets.

Summarizing the contributions of this paper is the following:

• RF is first applied to identify Mandarin tone in a speaker-independent manner.
• Through a large number of experiments, with three FFSs from only sound source features we find that RF for tone recognition is a high-stability, low-complexity approach.
• It is proven that the method proposed has good recognition for small sample sets and has strong generalization ability.

2. Materials and Methods

2.1. Data Description

Fundamental frequency, which varies greatly with speaker and gender, is an important feature parameter of tone recognition (the fundamental frequency range of males is 70~200 Hz and the fundamental frequency range of females voice is 140~400 Hz [13]). An effective speaker-independent Mandarin tone training system can be designed to first select gender and then start speech pronunciation practice. In this way, we need to study the relevant performance of the proposed algorithm in a certain gender, and the method for another gender can then be inferred.

In this work, the Syllable Corpus of Standard Chinese (SCSC) [14] was used to evaluate the effectiveness and robustness of our presented method. The corpus contains syllables used in daily Mandarin from fifteen male speakers named as m01, m02, …, m15, with the ages not noted. There are 1275 identical syllables per speaker and sound files are stored in high-quality 16 KHz sampling 16-bit data mono WAV format. In order to form the experimental speech dataset and balance the four tones, 40 monosyllables were selected from each speaker to obtain 600 syllables, including 10-tone 1, 10-tone 2, 10-tone 3, and 10-tone 4 (see Appendix A for details).

2.2. Preprocessing

In the short time frame processing, the frame length was set to 30 ms and frame shift was set to 10 ms. The double threshold method based on the short-time zero crossing rate and energy was then used to detect the voiced segments [15]. A Chebyshev low-pass filter with a cut-off frequency of 900 Hz was used to remove high frequency features from vocal tracts, and the auto-correlation method was used to derive fundamental frequency parameters [16].

2.3. Feature Fusion

After the preliminary experiment (see Section 3.2.1 for details), we found that cepstrum features have little contribution to improving the accuracy of tone recognition, so only sound source features should be used for feature fusion in tone recognition.

In this paper, seven original feature sets were used after reviewing the literatures, which are shown in

Table 1. The seven original feature sets.

Feature Set Name	Source	Features Number
S1	Reference [6]	22
S2	Reference [17]	13
S3	Reference [18]	6
S4	Reference [3]	16
S5	Reference [5]	18
S6	Reference [19]	7
S7	References [20,21]	12

.

Efficient RF models rely on high-quality FFSs. At present, it is not clear which features of the above feature parameters are important in the tone recognition task, so we used the three feature fusion methods shown in Figure 1 to explore this.

The specific details of these sets are as follows:

• FFS SI was directly composed of all 94 feature parameters from S1 to S7.
• The second method involved a BPNN, which has fine performance and wide application and was selected as a fixed classifier model to optimize the features of a tone recognition task. The number of nodes in the BPNN’s hidden layer was set to 32. The process was as follows: For each feature in SI, the ReliefF algorithm [22] was used to calculate the weight of each feature, and the weight was ranked from large to small. We then inputted the features into the BPNN in order for the purposes of tone recognition and stopped this process once recognition accuracy no longer rose. FFS SII was thus formed, which contained fifteen features.
• FFS SIII, which included twelve features, was obtained using the third method. Firstly, the top three feature sets of S1 to S7 were selected by the BPNN. Each feature from the top three sets was then ranked by ReliefF. Lastly, the twelve features could be optimized according to a process similar to that of the second method.

2.4. Classifier

The proposed method aims to handle small sample sets well, achieve fast modeling, and rely on simple calculations so that it may be deployed on small mobile terminals for Mandarin learners. Therefore, the tone recognition classifier should not be too complicated, suggesting low-complexity machine learning models to be more suitable for constructing the classifier. Back propagation neural networks, support vector machines, the Naive Bayes model (NBM), AdaBoost, and random forest are commonly used machine learning classifiers [6,23,24]. Therefore, the five classifiers mentioned above were used for the preliminary experiment in Section 3.2.2. We found that the RF classifier has better learning ability; however, no study has found that RF is suitable for Mandarin tone recognition.

2.5. Tone Recognition Classifier Based on Random Forest

2.5.1. CART Decision Tree and Random Forest Modeling

Random forest is composed of T decision trees, where T is the hyperparameter. The ID3, C4.5, and CART algorithms are commonly used decision tree algorithms [25]. It has been studied that the classification accuracy of the CART algorithm is better than that of other algorithms [26]. Thus, in this paper, the CART algorithm was used to construct the decision tree. The samples of each decision tree in the random forest are chosen randomly (reflected in the use of the bootstrap sampling algorithm to construct the sample set in the training process), which can effectively avoid overfitting and improve robustness.

In the training process, the shape of the training set is M₁*N, where M₁ denotes the number of training samples, which contains L types of tones, N denotes the number of features of each sample, and T is the number of decision trees.

The specific process is as follows:

• First, during the construction process of each decision tree, the training set is randomly selected and put back M₁ times to obtain a sample set with sample size M₁, where some data in the training set are selected multiple times and some are not. Thus, the M₁*N features matrix F = {f_i,j, i = 1,…, M₁, j = 1,…, N} is formed.
• Second, at the root node of each decision tree, one optimal feature with the smallest Gini index is selected from M₁*N features, and its feature value is the decision at the root node.

In the CART algorithm, the Gini index is used to select the node feature and represents the impurity of the dataset; the smaller the Gini index, the lower the impurity. This is expressed by the following equation:

$$Gini\left(D,F\right)=\frac{\left|D_1\right|}{\left|D\right|}Gini\left(D_1\right)+\frac{\left|D_2\right|}{\left|D\right|}Gini\left(D_2\right)$$

(1)

$$Gini\left(D_1\right)=1-{\sum }_{t=1}^S{\left(\frac{\left|S_t\right|}{\left|D_1\right|}\right)}^2$$

(2)

$$Gini\left(D_2\right)=1-{\sum }_{t=1}^S{\left(\frac{\left|S_t\right|}{\left|D_2\right|}\right)}^2$$

(3)

where D denotes the sample set in a certain node containing S tone types, whose number is |D|. The number of the tth tone type is |S_t|. D is divided into two subsets (D₁ and D₂) according to the value of the current node feature f_i,j, with D₁ including samples whose f_i,j feature value is lower than the value of the current node feature.

3.

In the next step, the matrix M1*N is divided into two parts: M₁₁*N and M₁₂*N. One optimal feature with the smallest Gini index is selected from M₁₁*N features, and its feature parameter is the decision at the branch node. A similar step is performed in M₁₂. M₁₁ is also divided into two parts, and the above process is repeated until all features are used or the tone type is provided as output, which is the leaf node.

4.

Repeat 1, 2, and 3 T times to construct T decision trees, which thus form a random forest. Figure 2 shows one decision tree.

2.5.2. Tone Recognition Based on the Random Forest Classifier

In the test process, the matrix of the test set is M₂*N, where M₂ denotes the number of test samples. The test set is input into the modeled random forest classifier, and the steps included are as follows:

• The test set is fed into the pre-trained random forest classifier.
• Starting from the root node of the current decision tree, the random forest classifier compares the feature parameters based on the value of the current node on each decision tree until the decision reaches the leaf node, which outputs the corresponding tone type.
• Since each decision tree is independent in the recognition process of each test sample, the final recognition result of the test sample is obtained via a voting process involving the results of multiple decision trees.

The entire training and test process is shown in Figure 3.

3. Experiment and Result

All experiments in this paper were implemented and tested on MATLAB R2020a using a 64-bit computer (Intel Core i7-12700 CPU, 2.10 GHz; 16.0 GB RAM).

3.1. Optimization Experiment of RF Classifier’s Hyperparameter T

Decision tree number T is an important hyperparameter in the RF classifier, and three RF classifiers were constructed based on SI, SII, and SIII. The number of T was set to 100, 200, 300, 400, and 500 in order to determine the optimal T value, and the variation in T values was reduced to 50 around the best value. The evaluation metrics were recognition accuracy rate (ACC), the area under the receiver operating characteristic (AUROC), and the area under the precision recall curve (AUPRC), which are commonly used for evaluating the performance of classifiers [27].

As shown in Figure 4 and

Table 2. Results of SI and RF.

Number of Decision Trees (T)	100	200	300	350	400	450	500
ACC (%)	98.17	98.00	98.00	98.00	98.33	98.00	98.00
AUROC (%)	98.79	98.69	98.68	98.68	98.88	98.68	98.68
AUPRC (%)	98.15	97.99	97.98	97.98	98.32	97.98	97.98

,

Table 3. Results of SII and RF.

Number of Decision Trees (T)	100	200	250	300	350	400	500
ACC (%)	97.00	97.17	97.33	97.33	97.50	97.17	97.33
AUROC (%)	98.02	98.12	98.23	98.23	98.35	98.12	98.23
AUPRC (%)	97.02	97.17	97.32	97.32	97.50	97.17	97.32

and

Table 4. Results of SIII and RF.

Number of Decision Trees (T)	100	200	300	350	400	450	500
ACC (%)	97.50	97.50	97.67	98.00	97.83	97.83	97.67
AUROC (%)	98.33	98.31	98.42	98.65	98.52	98.55	98.42
AUPRC (%)	97.47	97.48	97.63	97.97	97.80	97.81	97.63

the RF classifiers based on SI, SII, and SIII achieve the best performance at T = 400 (ACC, AUROC, and AUPRC are 98.33%, 98.88%, and 98.32%, respectively), T = 350 (97.50% ACC, 98.35% AUROC, and 97.50% AUPRC), and T = 350 (98.00% ACC, 98.65% AUROC, and 97.97% AUPRC), respectively.

3.2. Preliminary Experiment

3.2.1. Analysis of the Role of Vocal Tract Features in Tone Recognition

The feature parameters of the original speech signal can be divided into vocal tract features and sound source features. The former are mainly spectrum envelope parameters such as MFCC, and the latter are mainly time domain features, such as duration, energy, and fundamental frequency. MFCC, as a typical vocal tract feature, is commonly used in acoustic analysis such as automatic speech recognition [28] and dialect and language recognition [29]. However, the role of this feature in tone recognition needs to be evaluated through analysis experiments.

In the extraction of MFCC, preprocessing without 900 Hz low-pass filtering, fast Fourier transform (FFT) calculations, spectral line energy, Mel filtering energy, and a discrete cosine transform (DCT) cepstrum are needed. For each frame speech signal, twelve MFCCs and twelve ΔMFCCs were extracted, with the 24 feature parameters forming a one-dimensional feature vector. The ten frames of the central part of the syllable were selected for calculation, and these 240 feature parameters were named as the cepstrum feature set which was used for the tone recognition pre-experiment. When using five-fold cross-validation and a three-layer BPNN with 64 hidden layer nodes, tone recognition accuracy was 50.67%. It can be seen that the experimental result for tone recognition using the cepstrum feature set is not ideal. Next, we used both sound source features and cepstrum features for the experiment. We selected the fundamental frequency statistical features introduced in reference [6] as sound source features. Firstly, the fundamental frequency statistical features and cepstrum features were respectively used to carry out tone recognition experiments on the BPNN, and the two BPNN tone classifiers were then mixed and given weight α and 1-α, respectively, so as to explore the change in tone recognition accuracy with weight α. The specific realization formulas are as follows:

$$T_{pitch}^{*}=average\left\{T_{pitch}\left(\left.T_n\right|X_i\right)\right\}$$

(4)

$$T_{MFCC}^{*}={average\{T}_{MFCC}\left(\left.T_n\right|X_i\right)\}$$

(5)

where $T_{pitch}^{\mathrm{*}}$ and $T_{MFCC}^{*}$ are the recognized accuracy using the fundamental frequency statistical features and cepstrum features, respectively. n is the tone label, with values of 1, 2, 3, and 4. $T_{pitch}\left(\left.T_n\right|X_i\right)$ means the tone (T_n) accuracy of test sample set X_i (1 ≤ i ≤ N, N is the total number of samples) when using fundamental frequency statistical features and $T_{MFCC}\left(\left.T_n\right|X_i\right)$ means the T_n accuracy when using cepstrum features. The combined recognition accuracy with the two tone classifiers is defined as follows:

$$T^{*}=\alpha ·T_{pitch}^{*}+(1-\alpha )·T_{MFCC}^{*}$$

(6)

We carried out five-fold cross-validation and set the value of α as 0:0.1:1, and the results are shown in Figure 5.

It can be seen that the accuracy of tone recognition is not high when the cepstrum features are used alone. When the fundamental frequency statistical features and cepstrum features are used jointly for tone recognition, the fundamental frequency statistical features play a major role in greatly improving classification accuracy. Cepstrum features have little effect on improving the accuracy of tone recognition but greatly increase the operational complexity of parameter extraction. Therefore, cepstrum features are not used for tone recognition in this paper.

3.2.2. Analysis of Classifiers in Tone Recognition

The five classifiers under the typical structure were used for pre-experiments on S1 to S7, and the experimental results are shown in Figure 6. In S2, S5, S6, and S7, RF achieved the highest recognition accuracy, and in the remaining feature sets (S1, S3, and S4), the BPNN achieved the highest recognition accuracy, which indicates that the random forest has better learning ability. Further, the recognition accuracy of the five classifiers was averaged over seven feature sets. From the average accuracies, it is also evident that the RF classifier was slightly higher than that of others. The best result for the NBM could reach 96.67% and the worst was only 50.67%, which indicates that the NBM’s classification is either unstable or not robust; the accuracy of RF and the BPNN always remained above 95%. The classification results of the above five classifiers were obtained based on the same seven feature sets, and since different classifiers are based on different mathematical models, it shows the roles and effects of mathematical models in classifier modeling.

3.3. Comparative Experiment

In order to study the performance of the proposed method, comparative experiments were performed.

3.3.1. Comparative Experiments of Different Fusion Feature Sets

The four performance results were obtained by applying five-fold cross-validation on the three FFSs and RF classifiers. The results are shown in

Table 5. Recognition results from comparative experiments of different FFSs.

Set	SI	SII	SIII
ACC (%)	98.33	97.50	98.00
AUROC (%)	98.88	98.35	98.65
AUPRC (%)	98.32	97.50	97.97
APTPS (s)	0.0022	0.0011	0.0007

.

The Average Processing Time Per Sample (APTPS) index is the average time from extracting features to identifying a single syllable tone. From this index, it can be seen that the RF methods can achieve real-time identification and high recognition accuracy with three FFSs. It shows that the FFSs and RF tone recognition methods are highly efficient.

There are 94 features contained in SI, which is the most data contained in the three FFSs. This can describe the characteristics of tone more comprehensively; thus, the ACC, AUROC, and AUPRC values obtained are the highest and the classification effect is the best. Experimental results show that the classification effect of SII (15 features) is slightly worse than SIII (12 features), although the number of features for SII is more than SIII. This proves that simply increasing the number of features does not necessarily improve the recognition accuracy. This suggests that feature fusion is more important.

In addition, feature optimization is also needed to decrease algorithm complexity. By comparing SII and SIII, it can be seen that the classification effect is better and the running speed is faster with SIII. In general, the feature fusion method can not only reduce the running time, but also maintain high recognition accuracy.

3.3.2. Comparative Experiments of Small Sample Sets

In order to analyze the performance of the RF classifier in tone recognition, we carried out comparative experiments on small sample training sets. The experimental speech database (i.e., 600 syllables) was divided into 10 parts, with each part containing the same number of samples for the four tones. The proportion of training samples was reduced from 90% to 10% by step 10%. Consequently, the percentage of test samples was increased from 10% to 90% by the same step 10%.

Figure 7 shows the recognition results of the corresponding nine groups based on the three FFSs and RF classifiers. It can be seen that as the proportion of training samples decreases, tone recognition accuracy only decreases by about four percent. Nevertheless, even when training samples only accounted for ten percent of the database (i.e., 60 samples), there was still good accuracy of above 93.57%, which demonstrates the powerful learning ability of random forest.

4. Discussion

Using state-of-the-art methods (ToneNet [10], CNN [9]) as a benchmark, we compared accuracy, and the results are shown in

Table 6. Recognition performance of different methods on the SCSC database.

Method	ACC	Database	Suitable for Small Learning Terminal
ToneNet [10]	99.16%	SCSC	NO
CNN [9]	94.45%	SCSC	NO
The proposed	98.33%	SCSC	YES

. Although [10] used ToneNet to perform tone recognition on the SCSC database with an accuracy of 99.16%, it depends on a large amount of data and brings complex calculation, which is contrary to our intention to deploy a model on small mobile terminals. In [10], the authors used the CNN method of [9] on the SCSC, with results that were not as good as those obtained in this work. Therefore, considering operation time and computational complexity comprehensively, our method is the most cost effective.

In addition, we also carried out cross-dataset (i.e., beyond the original 600 syllables) testing experiments with balanced samples and extremely unbalanced samples. The balanced samples were 100 new syllables from the 15 original people, and were randomly selected from the SCSC, i.e., the samples of tone 1, tone 2, tone 3, and tone 4, with 25 of each. The extremely unbalanced samples were another 400 syllables from the same 15 people, and the ratio of tone 1, tone 2, tone 3, and tone 4 was 7:1:1:1, 1:7:1:1, 1:1:7:1, and 1:1:1:7, respectively. The balanced samples were respectively sent into three modeled FFSs and RF classifiers for tone recognition experiments, and the results are shown in Figure 8a. The extremely unbalanced samples were sent into the modeled RF classifier corresponding to SIII (which was the most cost effective), and the results are shown in Figure 8b. We find that tone 2 and tone 3 are more difficult to recognize than tone 1 and tone 4, both for balanced samples and for extremely unbalanced samples; this conclusion is consistent with that of previous research [30]. The test results of the cross-dataset experiments show that the tone classification algorithms based on RF have strong generalization ability.

5. Conclusions

This study introduces a novel Mandarin tone recognition approach which is based on random forest and feature fusion. Feature fusion and optimization can obviously reduce the complexity of an algorithm and are more suitable for portable Mandarin tone recognition. The random forest model is a robust classifier with low complexity in tone recognition. Through comparative experiments, the performance of RF on three different FFSs is validated, which shows that RF modeling has the advantages of high recognition accuracy, simplicity, and strong learning capability. This method can achieve good recognition effects based on the simplified FFS SIII using a small number of training samples. The proposed algorithm has achieved the expected results, but only simulation verification has been completed at present, and we will verify this on more databases and deploy the proposed method to the practical learning terminal in the future.