Evaluation of Speech Quality Through Recognition and Classification of Phonemes

Abstract

This paper discusses an approach for assessing the quality of speech while undergoing speech rehabilitation. One of the main reasons for speech quality decrease during the surgical treatment of vocal tract diseases is the loss of the vocal tractˈs parts and the disruption of its symmetry. In particular, one of the most common oncological diseases of the oral cavity is cancer of the tongue. During surgical treatment, a glossectomy is performed, which leads to the need for speech rehabilitation to eliminate the occurring speech defects, leading to a decrease in speech intelligibility. In this paper, we present an automated approach for conducting the speech quality evaluation. The approach relies on a convolutional neural network (CNN). The main idea of the approach is to train an individual neural network for a patient before having an operation to recognize typical sounding of phonemes for their speech. The neural network will thereby be able to evaluate the similarity between the patientˈs speech before and after the surgery. The recognition based on the full phoneme set and the recognition by groups of phonemes were considered. The correspondence of assessments obtained through the autorecognition approach with those from the human-based approach is shown. The automated approach is principally applicable to defining boundaries between phonemes. The paper shows that iterative training of the neural network and continuous updating of the training dataset gradually improve the ability of the CNN to define boundaries between different phonemes.

1. Introduction

One of the most common types of tumors of the speech-forming tract organs is cancer of the tongue [1]. Surgical treatment, consisting of a glossectomy [2], leads to the loss of part of the tongue involved in the formation of a number of phonemes. In particular, even during reconstructive surgery, a disruption of the tongue’s symmetry occurs. Ultimately, this leads to a disruption of the pronunciation, in particular, of pre-lingual consonants, which leads to a decrease in syllabic intelligibility and speech intelligibility in general.

In previous papers [3,4], we examined the application of automation in speech rehabilitation therapy for people who undergo a surgical intervention on speech organs. Currently, the evaluation of speech quality during rehabilitation is given by several experts (speech therapists) for providing more objective results. This procedure is time-consuming for the experts and requires a patient to come to a hospital. It is not always convenient and possible for a patient. Due to these problems, the idea of automating the rehabilitation process came up.

A speaker-dependent neural network is trained for each patient on audio records pronounced by a patient before a surgery. The neural network learns how a patient pronounces each type of phoneme, thus aiming to recognize whether the patient pronounces phonemes in the same manner after the surgery. Speech recorded by a patient is a set of syllables which includes the most problematic phonemes, such as /k/, /s/, /t/, etc. (hereinafter referred to as problematic phonemes). The list of these phonemes was compiled at the first stage of the study [4]. We could potentially use the complete classical table of syllables from GOST R 50840-95 [5] (5 tables, 250 syllables according to the method of evaluating syllable intelligibility). However, recording 250 syllables per session is a tedious task for a patient. Therefore, in agreement with physicians engaged in speech rehabilitation, it was decided to limit the complete list to 90 syllables. The sample is oriented towards the main problematic phonemes (/k/, /s/, /t/ and their soft implementations). The most problematic phoneme /r/ was excluded from consideration, because the mechanism of producing this phoneme changes fundamentally after the operation. Consequently, its comparison with the standard is meaningless. The audio records obtained before a surgery were considered as a benchmark. Subsequently, the quality of the speech after the surgery was evaluated in comparison with this benchmark.

In order to train the neural network to recognize phonemes in after-surgery records, it was necessary to know the start and the end point of each phoneme in before-surgery records. The phonemic composition of all syllables was known. However, the start and the end points of each phoneme were not defined. Consequently, this raised the question of phoneme alignment.

Initially, audio records were segmented at a phoneme level manually. The start and the end points of each phoneme in a syllable were defined acoustically and visually using a software tool that displayed spectrograms. For manual segmentation into phonemes and spectrograms, the software Praat [6] and Wavesurfer [7] were used. This approach was quite time intensive. As a result, it was necessary to move on to an automatic segmentation.

The aim of this work was a reduction in the time taken to align phonemes manually in before-surgery audio records.

2. Evaluation of Speech Quality Through Recognition and Classification of Phonemes

2.1. Methods of Syllable Recognition

The task of recognizing syllables and phonemes is part of a more general task of speech recognition. To solve it, when conducting hierarchical analysis (from the level of phonemes and syllables to the level of sentences and their groups), the following approaches can be used:

• hidden Markov models [8,9];
• hidden Markov models and Gaussian mixture models [10,11];
• deep learning neural networks [12,13];
• different hybrid models [14,15,16].

The second approach is end-to-end speech recognition. It differs from sequential hierarchical analysis in that it allows you to analyze the original signal and move to higher levels of analysis (for example, the level of words), bypassing lower levels [17,18].

In this paper, the recognition of phonemes within isolated syllables without context is of interest. This makes adjustments impossible at the upper levels of analysis. A statistical imbalance in the phonetic material increases the share of “problem” phonemes and contributes to the impossibility of using ready-made speech models. This makes it impossible to use ready-made solutions in the field of speech recognition, such as the Hidden Markov Model Toolkit (HTK) [19], Kaldi [20,21], Sphinx [22], and their local-language analogues.

In assessing the quality of syllable pronunciation, the following options may be identified:

-

the direct recognition of pronounced phonemes within a syllable. The main disadvantage of this approach is a large number of classes (which is equal to the number of phonemes). As a result, we have a high level of error. On the other hand, the positive aspects include the straightforward result in the form of a phoneme sequence and the ease of error determination;

-

the recognition of phonemes within a syllable as instances of classes formed of phonetic groups. The positive side is that the accuracy increases due to the reduced number of classes. However, it leads to a lack of direct interpretation, hence a lack of direct evaluation of the assessment according to GOST R 50840-95;

-

the identification of boundaries between phonetic segments using a neural network for the follow-up application of parametric approaches in comparing these segments. In this case, the accuracy of determining transitions between phonemes is more important than the classification accuracy. A measure of difference between the selected segments is determined on the basis of previously developed parametric methods [3]. The disadvantage of this option is the unavailability of quantitative assessment such as the classic syllable intelligibility (as defined in Section 2.1) in the output of the system.

2.2. Direct Recognition of Phonemes

In the context of recognition, we used an approach for searching audio fragments similar to those in the training dataset.

The approach implemented within the framework of this task had several limitations, some of which were introduced artificially.

• Dependence on a speaker. A model for assessing the quality of speech was built for each new speaker. There was no task to improve the quality of speech in relation to the already established manner of pronouncing phonemes or to the presence of speech defects. The task in the rehabilitation process was to maximize the conformity of a patientˈs after-surgery speech with their speech before the operative treatment. This limitation significantly simplifies the task because there is no need to use a large database of records from many speakers for training.
• Limited number of phonemes. We were primarily interested in the quality of pronouncing the phonemes that were most susceptible to change after the operation. For this reason, the table of syllables focuses specifically on those problematic phonemes.
• The assessment speed of one syllable, pronounced by a patient while practicing at the rehabilitation stage, should be as fast as possible. Currently, the evaluation takes 3 seconds per syllable. The training time of the convolutional neural network (CNN) takes less than one hour. The Adam optimizer with a mini-batch size of 128 was used for training of the neural network [23]. However, the training time did not matter much since the period of time between a before-surgery session and the first rehabilitation session was approximately one week.
• Within the framework of the paper, the term “syllable intelligibility” refers to the proportion of correctly recognized syllables among all of them pronounced by a patient in accordance with a predefined set of syllables. In the future, values of the output layer of the neural network will be used to assess the degree of similarity between a pronounced phoneme and the correct one in order to implement the biofeedback mechanism in the rehabilitation process. However, in this paper, the idea was to prove the applicability of this automated speech quality assessment approach to speech rehabilitation.
• It was known in advance which syllable was pronounced. There was no need to interpret the sequence of recognized phonemes, transforming it into a syllable; it was only necessary to estimate the proportion of correctly pronounced phonemes in that sequence.

To implement a deep neural network for the recognition of syllables in the framework of assessing the quality of their pronunciation, the computing environment MATLAB 2018a (MathWorks, Natick, MA, USA) [24] containing the Neural Network Toolbox package was used. The internal architecture of the neural network (30 layers) was chosen based on recommendations of the MATLAB test pattern for command recognition. The set of the CNN was as follows: the input layer, 2 × (Convolutional Layer, Batch Normalization Layer, Rectified Linear Unit Layer, Max Pooling Layer), 2 × (Dropout Layer, Convolutional Layer, Batch Normalization Layer, Rectified Linear Unit Layer), Max Pooling Layer, 2 × (Dropout Layer, Convolutional Layer, Batch Normalization Layer, Rectified Linear Unit Layer), Max Pooling Layer, Fully Connected Layer, Softmax Layer, and Weighted Cross EntityLayer. The outputs had the following structure: vocalization output, softness output, and 21 classes for the phoneme identification, for a total of 23 outputs. The input layer contained 4800 neurons.

2.3. Algorithm of Automatic Time Alignment at Phoneme Level

First of all, the speaker-independent neural network was trained on a Russian language audio dataset containing 7 sentences recorded by 10 speakers of both genders. Every audio file was complemented by a transcript describing its phonemic composition and time periods.

The algorithm to align an unsegmented audio file included the following steps:

• Training a neural network (initially on data from an audio-aligned corpus).
• Recognition of a phonemic composition as phonetic sequences for each syllable.
• Determining time-aligned phonemic transcriptions of syllables.
• Adjustment of recognized phonemic compositions and selection of syllables with a correctly defined phonemic composition.
• Forming additional data from transcriptions of correctly recognized syllables.
• Going to step 1 and retraining the neural network on an updated dataset (including new data formed at step 5).

The steps were repeated until the required level of quality was reached.

The main sequence of actions in IDEF0 notation [25] is shown in Figure 1.

2.3.1. Phoneme Recognition and Time Alignment

At the data preprocessing stage, WAV format audio files at a sample rate of 16 kHz were sliced into overlapping 20 ms frames with a frame step of 1 ms. Mel-frequency cepstral coefficients (MFCCs) were extracted for each frame.

One audio file of a syllable record lasts about 1000 ms (1 s). Thus, we obtained 981 frames of 20 ms length each. The arg max rule was used to compute the label for each time step. We referred to the independent labelling of each time step, or frame. Figure 2 depicts the best path decoding example [26,27] for a 1 s audio file.

As a result, we obtained a sequence of labels. Many labels were certainly repetitive because a phoneme lasted a few milliseconds and frames were overlapped. The algorithm of removing the duplicates and calculating the phoneme’s start and end points was considered further.

We defined two parameters for this algorithm:

• The minimum length of a phoneme (the minimum number of consecutive frames labeled as the same phoneme) as min_seq_len.
• The maximum length of deviations from a consecutive sequence of the same phoneme labels as max_dev_len.

These two parameters might vary depending on the type of phoneme. For example, phonemes /g/ and /d/ usually lasted less than phoneme /t/ or /s/.

Considering a simplified sequence shown in Figure 3, which contains only 30 labels, the result of applying the algorithm described further (below Figure 3) is a time-aligned phonemic transcription as presented in

Table 1. Time-aligned phonemic transcription of the syllable [ɡˈɨs].

No.	Phoneme	Start	End
1	pause	1	6
2	g + gˈ	7	12
3	vow	16	20
4	s + sˈ	21	25
5	pause	26	30

.

If we suppose max_dev_len = 1 and min_seq_len = 5 for all the phonemes, then:

• SEQUENCE_1. 1:pause/ 2:pause/ 3:pause/ 4:pause/ 5:s + sˈ (dev_len = 1)/ 6:pause/ 7:g + gˈ (dev_len = 2 > max_dev_len, seq_len = 6 ≥ min_seq_len) ≥ “pause 1–6” is appended to the result
• SEQUENCE_2. 7:g + gˈ/ 8:g + gˈ/ 9:g + gˈ/ 10:g + gˈ/ 11:g + gˈ/ 12:g + gˈ/ 13:d + dˈ (dev_len = 1)/ 14:d+dˈ (dev_len = 2 > max_dev_len, seq_len = 6 ≥ min_seq_len) ≥ “g + gˈ 7–12” is appended to the result
• SEQUENCE_3. 13:d + dˈ/ 14:d + dˈ/ 15:d + dˈ/ 16:vow (dev_len = 1)/ 17:vow (dev_len = 2 > max_dev_len, seq_len = 3 < min_seq_len) ≥ nothing is appended to the result
• SEQUENCE_4. 16:vow/ 17:vow/ 18:vow/ 19:vow/ 20:vow/ 21:s + sˈ (dev_len = 1)/ 22:s + sˈ (dev_len = 2 > max_dev_len, seq_len = 5 ≥ min_seq_len) ≥ “vow 16–20” is appended to the result
• SEQUENCE_5. is formed in the same way as SEQUENCE_4. “s + sˈ 21–25” is appended to the result
• SEQUENCE_6. 26:pause/ 27:pause/ 28:pause/ 29:pause/ 30:pause (last frame reached, seq_len = 5 ≥ min_seq_len) ≥ “pause 26–30” is appended to the result

2.3.2. Adjustment of Recognized Phonemic Composition

Since time-aligned phonemic transcriptions were defined for all the syllables, it was possible to apply some operations to adjust partly incorrect phonemic compositions of some syllables. We proposed three operations for the adjustment:

• Substitution
• Shift
• Removal

These operations are simple and intuitive, and applied as shown in Figure 4.

If a phonemic composition of a syllable became completely correct after the adjustment, the time-aligned phonemic transcription of this syllable was used to create new data for further neural network training.

3. Results of Experiments

3.1. The Direct Recognition of Problem Syllables

At this stage, a phoneme was considered to be correctly recognized if more than 50% of the correct samples were present. Results of the assessment of syllable intelligibility by five experts and using the proposed approach are presented in

Table 2. The results of the assessment of syllable intelligibility by experts and using the proposed approach based on CNN for healthy speakers with and without the use of a tongue and for patients before and after surgery.

Person No.	Normal by the Experts	Without Tongue by the Experts	Normal by the CNN	Without Tongue by the CNN
Person1	1.00	0.43	0.64	0.24
Person2	1.00	0.43	0.43	0.20
Person3	1.00	0.57	0.50	0.29
Patient No.	Before-Surgery by the Experts	After-Surgery by the Experts	Before-Surgery by the CNN	After-Surgery by the CNN
Patient1	1.00	0.44	0.53	0.21
Patient2	0.99	0.68	0.47	0.31
Patient3	1.00	0.57	0.56	0.20

. The recognition of the whole syllable with problematic phonemes was considered. Estimation with its standard deviation by five different experts and individual neural networks for every person/patient is presented. The group of patients and the group of healthy speakers included three people in each group. The healthy speakers spoke with and without the use of their tongues in order to imitate the pronunciation of patients before and after surgery accordingly. The patients made their audio records before and after undergoing the surgery. “Person No.” are healthy speakers and “Patient No.” are patients who began the rehabilitation. “Normal” is a standard speech for healthy speakers. “Before-surgery” is a standard speech before operation for patients. “Without tongue” is speech without the use of a tongue for healthy speakers. “After-surgery” is speech after operation for patients. Records contain syllables with problematic phonemes (/t/, /k/, /s/, /tˈ/, /kˈ/, /sˈ/ [3]). The list of audio records contains 90 syllables.

Table 3. The results of the assessment of syllable intelligibility by experts and using the proposed approach based on CNN for healthy speakers with and without the use of a tongue and for patients before and after surgery. The recognition of only problematic phonemes from syllables is considered.

Person No.	Normal by the Experts	Without Tongue by the Experts	Normal by the CNN	Without Tongue by the CNN
Person1	1.00	0.43	0.95	0.60
Person2	1.00	0.43	0.97	0.51
Person3	1.00	0.57	0.97	0.65
Patient No.	Before-Surgery by the Experts	After-Surgery by the Experts	Before-Surgery by the CNN	After-Surgery by the CNN
Patient1	1.00	0.44	0.99	0.56
Patient2	0.99	0.68	0.97	0.69
Patient3	1.00	0.57	0.99	0.59

contains the same information, but for the calculation of the scores only problematic phonemes were used instead of the whole phoneme composition of syllables. This nuance is not substantial for experts, but has an important influence on the neural network. The main reason for this fact is the larger number of problematic phonemes in the training dataset in comparison with other phonemes. As a result, the neural network has a much smaller number of errors with respect to problematic phonemes.

After considering Table 2 and Table 3, the following conclusions were drawn.

• In Table 2, even for a healthy speaker, the syllable intelligibility calculated by the CNN did not reach 100%, thus diverging from the opinion of the experts. However, mistakes mostly arose from “non-problematic” phonemes, which is explained by their small share in the syllable set table. In particular, some of the phoneme implementations in the table are missing, since recognition was not the ultimate goal of the system.
• On the other hand, for problematic phonemes, the results of which are presented in Table 3, the difference is statistically insignificant when using the Studentˈs t-test with the 0.95 significance level. In the future, it is possible to increase this value due to the variation in the structure of the neural network used and its adaptation to the problem being solved.
• The qualitative assessment of syllable intelligibility given by the CNN corresponds to the experts’ ones. This fact allows a discussion about the applicability of the proposed approach for solving the problem of speech quality assessment during speech rehabilitation. It also confirms the consistency at the level of ranking positions between the classical expert method for estimating syllable intelligibility and the proposed method using neural networks.

3.2. The Speaker-Dependent Neural Network for Class Segmentation

Retraining the neural network on an updated dataset makes it more and more speaker-dependent as well as more precise for a certain speaker-patient.

Figure 5 depicts the graphical representation of gradual changes in a phonemic composition of syllable [ɡˈɨs] recognized by the neural network.

Among the characteristic features, the decrease in phonemes belonging to the class [d + dˈ] in the classification results can be distinguished. The count of windows with this class decreases with an increase in the number of iterations. This fact can be explained by the relatively small representation of the given class in the general sample. As a result, with additional training, fewer values fall into the training set, which leads to a decrease in the detection of this class. If we compare this result with the set of sounds [g + gˈ] and [k + kˈ] when solving the ultimate problem of setting boundaries with the known phonetic composition of the syllable and an a priori representation of the class, the following dependence is visible. The significance of the class that is absent in the identified syllable decreases with an increase in the number of iterations.

The experiment of syllable segmentation at a phoneme level was conducted with following characteristics:

• 1 initial speaker-independent neural network trained on 7 sentences recorded by 10 speakers;
• 6 Russian-speaking patients (3 female and 3 male voices);
• 8 classes of phonemes were considered: [ch + sh], [d + dˈ], [g + gˈ], [k + kˈ], [pause], [s + sˈ], [t + tˈ], [vow];
• a set of 15 syllables (see

  Table 4. Syllable set.

  No.	Serial Number	Syllable (in accordance with the International Phonetics Association (IPA) [28])	Phonemic Composition (in accordance with 8 classes)
  1	265	[kˈasʲ]	[k + kˈ, vow, s + sˈ]
  2	271	[dˈokt]	[d + dˈ, vow, k + kˈ, t + tˈ]
  3	282	[kʲˈæsʲtʲ]	[k + kˈ, vow, s + sˈ, t + tˈ]
  4	283	[kʲˈɵs]	[k + kˈ, vow, s + sˈ]
  5	289	[ʂkʲˈet]	[ch + sh, k + kˈ, vow, t + tˈ]
  6	295	[sˈosʲ]	[s + sˈ, vow, s + sˈ]
  7	298	[sˈɨt͡ɕ]	[s + sˈ, vow, ch + sh]
  8	305	[dˈɨs]	[d + dˈ, vow, s + sˈ]
  9	306	[ɡˈɨs]	[g + gˈ, vow, s + sˈ]
  10	310	[sʲˈit͡ɕ]	[s + sˈ, vow, ch + sh]
  11	311	[sʲˈesʲ]	[s + sˈ, vow, s + sˈ]
  12	316	[ksˈɛt]	[k + kˈ, s + sˈ, vow, t + tˈ]
  13	323	[ʂˈɨsʲ]	[ch + sh, vow, s + sˈ]
  14	332	[stˈɨt͡ɕ]	[s + sˈ, t + tˈ, vow, ch + sh]
  15	351	[t͡ɕˈætʲ]	[ch + sh, vow, t + tˈ]

  ).

Table 5. Summary of applying the algorithm of automatic syllable segmentation at a phoneme level.

Patient No.	Gender	Fully Correct Recognized Syllables	Problematic Syllables	Number of Iterations
1	female	13/15	[dˈokt], [stˈɨt͡ɕ]	6
2	female	12/15	[kˈasʲ], [dˈokt], [ʂkʲˈet]	6
3	female	12/15	[ʂkʲˈet], [ɡˈɨs], [stˈɨt͡ɕ]	5
4	male	12/15	[dˈokt], [ksˈɛt], [stˈɨt͡ɕ]	2
5	male	12/15	[dˈokt], [ɡˈɨs], [sʲˈit͡ɕ]	4
6	male	13/15	[ksˈɛt], [stˈɨt͡ɕ]	5

presents the results obtained for each patient. The neural network was retrained as many times as it needed to recognize an accurate phonemic composition for at least 12 out of 15 syllables.

After considering the results in Table 5, the following conclusions were drawn.

• The proposed algorithm gives the expected result and may be applied for automatic identification of a phonemic composition of syllables as well as for determining the start and the end time points for each phoneme.
• The syllables [dˈokt] and [stˈɨt͡ɕ] turned out to be the most problematic syllables. This may be due to two consecutive voiceless phonemes, such as /k/, /t/ for [dˈokt] or /s/, /t/ for [stˈɨt͡ɕ].
• The efficiency of the algorithm may be improved by fine-tuning min_seq_len and max_dev_len parameters based on the specificities of phonemes.

4. Conclusions

In this paper, we proposed an approach to speech quality assessment based on a convolutional neural network. In the context of this work, speech recognition was applied for estimating syllable intelligibility according to the method presented in GOST R 50840-95 “Speech transmission over various communication channels”. Methods for assessing the quality of the recognition were considered. In this approach’s framework, the final deep neural network can act as an auditor and issue an appropriate quantitative estimate at the output. The received values allow a discussion about the absence of obvious contradictions between the results from the CNN and the estimations provided by experts. In addition, for objective assessments made by humans, it was necessary to have the opinions of five experts. This significantly reduces the practical applicability of the method with experts’ direct participation. The use of a neural network instead of experts solves this problem.

It is also possible to formulate several points for a more accurate study of the proposed approach in order to improve the obtained results, ensure additional confirmation of their reliability, and confirm the approach’s implementation within the version of the speech quality assessment system used during speech rehabilitation.

• Verification of the system’s functioning using several trained neural networks that can act as separate auditors [29] (in accordance with GOST R 50840-95, the plan is to use five neural networks).
• Use of a fraction of correctly recognized phonemes in a time interval, as well as use of quantitative outputs of a neural network to increase the flexibility of the values obtained, currently at the level of a correctly/incorrectly recognized syllable.
• Verification of the obtained approach on the full extent of the available data for the rehabilitation process with real patients.

For the task of classification and segmentation:

• Recurrent training with the addition of data allows researchers to increase the accuracy of speech information analysis.
• The total classification accuracy without reference to the speaker exceeds 82%.
• The number of training iterations, at which an increase in the resulting accuracy is recorded, does not exceed six cycles of additional training.

According to the results, the approach for obtaining parametric estimates after segmentation on the basis of the classification of the phonemes of syllables is further proposed.