Whispered Speech Recognition Based on Audio Data Augmentation and Inverse Filtering

Abstract

Modern Automatic Speech Recognition (ASR) systems are primarily designed to recognize normal speech. Due to a considerable acoustic mismatch between normal speech and whisper, ASR systems suffer from a significant loss of performance in whisper recognition. Creating large databases of whispered speech is expensive and time-consuming, so research studies explore the synthetic generation using pre-existing normal or whispered speech databases. The impact of standard audio data augmentation techniques on the accuracy of isolated-word recognizers based on Hidden Markov Models (HMM) and Convolutional Neural Networks (CNN) is examined in this research study. Furthermore, the study explores the potential of inverse filtering as an augmentation strategy for producing pseudo-whisper speech. The Whi-Spe speech database, containing recordings in normal and whisper phonation, is utilized for data augmentation, while the internally recorded speech database, developed specifically for this study, is employed for testing purposes. Experimental results demonstrate statistically significant improvement in performance when employing data augmentation strategies and inverse filtering.

1. Introduction

Human communication mostly relies on speech, playing a significant role in our everyday lives. Unlike traditional tactile interfaces, such as the mouse and keyboard, which require the user’s full attention, speech-based human-machine interaction offers a more natural and intuitive mode of communication [1].

Automatic Speech Recognition (ASR) employs various complex computational algorithms to transcribe spoken words into text. ASR systems are distinguished by several key characteristics [2]:

• Initial user training is unnecessary, as speech is a natural skill that develops from a young age.
• Speaking enables communication up to ten times faster than typing or writing.
• The users can be partially engaged in other activities, as their hands and eyes are not occupied.
• Microphones are affordable and have smaller dimensions compared to keyboards.

Modern ASR systems demonstrate notable speed and accuracy, surpassing human accuracy in specific controlled environments for Large Vocabulary Continuous Speech Recognition (LVCSR) [3]. Nevertheless, these systems exhibit a high degree of sensitivity when faced with speech that differs from the one used in training, typically consisting of normal speech recorded under controlled conditions in a laboratory setting. Recognizing such unconventional speech, including changes in vocal effort, variations in speaker emotional states and accents, and adverse conditions such as background noise, reverberation, and loudness, presents a challenging task for researchers in the field.

Speech can be classified into five modes based on the level of effort: whispered, soft, normally phonated (normal), loud, and shouted speech [4]. Whisper is the most unique mode due to the absence of glottal vibrations and noise excitation of the vocal tract. People tend to whisper or lower their voices for various reasons. Firstly, whispering is used in situations where normal speech is not allowed or deemed inappropriate, such as in a theater or reading room. Secondly, it is employed to convey confidential information that uninvolved individuals should not hear. Lastly, whispering is utilized in criminal activities to conceal one’s identity. Apart from intentional whispering, it can also occur as a result of health issues (laryngitis or rhinitis) [5].

Whispered speech poses the greatest challenge to the ASR system’s performance compared to other speech modes. The notable acoustic mismatch between normal and whispered speech plays a key role in such performance degradation. To improve the robustness, an extensive speech database containing different speech modalities and variability should be developed, especially for modern deep-learning-based ASR systems [6]. However, this procedure is expensive and time-consuming. To address these constraints and enhance the generalization capability of ASR systems, data augmentation (DA) methods have been utilized. A review of the augmentation techniques used in the literature for speech recognition tasks is provided in

Table 1. The review of augmentation techniques.

Ref.	Augmentation	Metric	Speech
[7]	CycleGAN, SpecAugment, Speed perturbation	Word Error Rate (WER)	Children’s
[8]	Glottal source extraction, Silence removal, Impulse response, Noise addition	Unweighted Average Recall (UAR)	Emotional
[9]	GANs, Time Stretch, Pitch Shift, Add Noise	UAR, F-score	Emotional
[10]	Speed perturbation	WER	Continuous
[11]	Simulated Room Impulse Response, Add noise	WER	Continuous
[12]	MP3 and Full rate GSM	WER and Mean Opinion Score (MOS)	Conversational
[13]	Add noise	WER	Continuous
[14]	Multiplying the region, Replacing the region, Input concatenation	WER	Read
[15]	CycleGAN	WER	Whispered

.

The application of data augmentation in audio classification tasks is discussed in [16]. Study [17] demonstrates the synthesis of speech waveforms based on Continuous Wavelet Transform (CWT), which produces natural-sounding synthetic speech. Recent advancements in whispered speech recognition involve pseudo-whispered speech conversion through the following three steps [18]:

(1)

Removing the glottal contribution via spectral subtraction;

(2)

Shifting the first formant using frequency warping;

(3)

Increasing the formant bandwidth with a moving average filter.

Audio data augmentation utilizes various algorithms to generate new (artificial) audio samples from existing ones (natural) to enlarge the dataset size and mitigate over-fitting issues in ASR systems.

In the context of low-resource languages such as Serbian, where only a limited number of publicly available speech databases are accessible, utilization of audio data augmentation can play an important role. The main motivation of this research study is to demonstrate that by applying audio data augmentation techniques to some existing datasets, we can elevate the quality and diversity of training data, ultimately leading to improved performance of ASR.

The primary contributions outlined in this paper are:

• The whispered speech database, which has been annotated and made publicly accessible, was developed for testing purposes.
• The efficacy of using inverse filtering as an algorithm for data augmentation in generating pseudo-whisper has been validated in speaker-independent whisper recognition.
• The efficacy of typical augmentation techniques (such as pitch shifting, time stretching, and volume adjustment) in artificially enlarging whispered speech databases has been illustrated for an ASR system utilizing Hidden Markov Models (HMM) and Convolutional Neural Networks (CNN).
• The beneficial effect of parallel augmentation, where a single natural utterance produces multiple artificial ones, has been verified to enhance the performance of CNN-based recognizers.

This study analyzes the performance of the recognition systems that utilize Mel Frequency Cepstrum Coefficients (MFCC) as feature vectors. Additionally, Linear Predictive Coding (LPC), based on the source-filter model of speech signals, is commonly employed in the formant extraction procedure. In this paper, LPC coefficients are utilized for inverse filtering to generate pseudo-whisper utterances from normal speech. Stochastic Gradient Descent (SGD) is an effective algorithm for training neural networks, and the CNN recognizer developed in this study uses SGD with an optimization algorithm based on Adaptive Moment Estimation (Adam).

The paper is structured into seven sections as follows. Section 2 provides an overview of whispered speech and compares it with normal speech. Section 3 explores the methodology and theory underlying audio data augmentation techniques. Section 4 explains the rationale behind the usage of the inverse filtering algorithm for artificially expanding the speech database used for training. Section 5 details the experimental preparation, including the speech databases (natural and augmented) used for training and testing, the feature extraction procedure, and the characteristics of ASR systems. Section 6 presents the experimental results and discussion, focusing on the contribution of augmentation strategies and inverse filtering concerning WER reduction. Finally, Section 7 concludes with final remarks and outlines directions for future work.

2. Whispered Speech

Whisper represents a unique mode of verbal expression that, based on its distinct characteristics, nature, and way of production, deviates considerably from normal speech. As previously stated, the key features of whispering include the lack of a fundamental frequency and the noisy excitation of the vocal tract. Formant frequencies of whispered vowels are significantly greater than those of the neutral voice [19]. Compared to normal speech, whispered speech exhibits reduced frame energy, extended speech and silence durations, a flatter long-term spectrum, and a lower Sound Pressure Level (SPL).

Even though a greater level of concentration is required for understanding whispered speech, its intelligibility remains remarkably high. An average identification accuracy of 82% in identifying vowels within [hVd] syllables when spoken in a whisper is demonstrated in [20]. Conversely, non-verbal cues such as age, sex, emotional state, and identity are often difficult to discern in whispered speech.

Figure 1 illustrates the waveform and spectrogram of the phrase “Govor šapata” (“Whispered speech” in English) in Serbian. A phonetic transcription is also provided for normal speech (in capital letters) and whisper (in small letters). The amplitude levels in Figure 1a display a noticeable difference between the two modes of speech due to a loss of sonority. Additionally, the spectrogram in Figure 1b reveals the following characteristics:

• Fricatives such as /š/ (i.e., /ʃ/ in IPA notation) and plosives /p/ and /t/ are well preserved in whisper.
• The vibrant /r/ exhibits a similar shape in both normal and whispered speech.
• The absence of vertical lines representing glottal pulses shows that a harmonic structure of vowels is not present in a whisper.

3. Audio Data Augmentation

Data augmentations encompass a variety of effective algorithms developed to generate synthetic data and modify existing datasets to improve recognition systems’ capability for generalization. Even though the technique was initially created for image processing, audio data may as well benefit from its usage. In the area of audio-related machine-learning tasks, a number of audio augmentation techniques have been created. Depending on the format of the input audio, data augmentation techniques can be categorized into two groups:

• Audio data augmentation and
• Spectrogram audio data augmentation.

Audio Data Augmentation (Audio DA) includes a set of modifications (transformations) performed directly on the raw audio signal (waveform). Commonly used augmentation techniques are pitch shifting, time stretching, volume control, adding noise, and time shifting.

• Pitch shift

The voice fundamental frequency (F₀) is one of the speaker’s most significant individual acoustic characteristics. This method modifies the F₀, usually expressed in semitones (one-twelfth of an octave). Pitch shifting is widely recognized as one of the most common augmentation techniques.

• Time stretch

This method adjusts the speaking rate or speed of the speech signal while maintaining the pitch. The speedup factor, also known as the stretch ratio, typically ranges from 0.8 (slower) to 1.2 (faster).

• Volume control (volume gain)

This method applies a gain to the speech signal, typically measured in decibels.

• Add noise

This technique adds white noise to the speech signal based on the specified Signal-to-Noise Ratio (SNR) as defined by Equation (1), where $P_s$ represents the power of the signal and $P_n$ represents the power of the noise.

$$SNR\left[\mathrm{d}\mathrm{B}\right]=10{log}_{10}{\displaystyle \frac{P_S}{P_N}}$$

(1)

Despite improving its robustness in noisy conditions, it may not be effective when the test set is noiseless [21].

• Time shift

This method shifts the audio signal left or right by a specified time in milliseconds. While it is more useful in sound classification tasks, its applicability in ASR tasks is limited [21].

The other audio DA techniques include:

• Wow Resampling
• Clipping
• Add Impulse Response
• Filtering
• MP3 Compression
• Inversion

The image and spectrogram DA techniques are described in more detail in [22]. Traditional methods used for data augmentation include flip, zoom range, shift, rotation angle, and brightness range. Spectrogram DA is applied to spectral images and is often used in audio classification tasks.

Audio augmentation algorithms can be implemented either in a sequential manner (in series) or independently (in parallel). The architecture of sequential augmentation is illustrated in Figure 2a (single augmented output) and Figure 2b (N augmented output signals) for parallel random sequential augmentations. The probability of applying an augmentation and the value of any parameters that are probabilistically determined are independent.

4. Inverse Filtering

Inverse filtering is a powerful technique in speech processing used to isolate the excitation component from the vocal tract shape. This chapter focuses on using inverse filtering to transform normal speech into a pseudo-whisper by removing the formant structure of voiced sounds and flattening the spectrum [23,24,25,26].

Linear Predictive Coding (LPC) [27] is a method that approximates the spectral envelope of a speech signal using a finite number of coefficients. The speech signal (s[n]) can be modeled as a linear combination of its past (p) samples plus an error term (e[n]):

$$s\left(n\right)=\sum _{i=1}^p{a}_{i}s\left[n-i\right]+e\left[n\right],$$

(2)

where:

• s[n] is the current value of the signal;
• $a_i$ are the coefficients of the model;
• s[n − i] are the previous values of the signal up to order p;
• e[n] is the error term or white noise at time n.

LPC coefficients can be calculated using different methods, including the autocorrelation method and Burg’s method [28]. Burg’s method estimates LPC coefficients by minimizing the forward and backward prediction errors while ensuring the model’s stability.

The vocal tract is modeled as an all-pole filter, represented as a transfer function H(z) in the z-domain, where $a_i$ are the coefficients and p is the order of the system:

$$H\left(z\right)={\displaystyle \frac{1}{1-\sum _{i=1}^{\mathrm{p}}{a}_i{z}^{-i}}}$$

(3)

This filter captures the resonance characteristics or formants of the vocal tract [29,30]. The inverse filter is designed to remove the effects of the vocal tract filter:

$$IF\left(z\right)=1-\sum _{i=1}^{\mathrm{p}}{a}_i{z}^{-i}$$

(4)

By applying this filter to the original signal, the formant structure is removed, resulting in a flattened spectrum that emphasizes the excitation signal. From a signal processing viewpoint, the steps include frame blocking and windowing. First, the speech signal is divided into overlapping frames (25 ms with 10 ms overlap). Then, a window function (Hamming) is applied to each frame. For each frame, the LPC coefficients are computed using Burg’s method. The inverse filter IF(z) is then applied to each frame:

$$e\left[n\right]=s\left[n\right]-\sum _{i=1}^{\mathrm{p}}{a}_{i}s[n-i]$$

(5)

Filtering can be conducted in both the time domain and the spectral domain. In this study, the signals were convolved directly with the inverse filter in the time domain. Alternatively, in the spectral domain, the signals should be transformed to the frequency domain using FFT, multiplied with the inverse filter, and then transformed back.

The order (p) of the LPC filter affects the accuracy of the vocal tract model and the resulting spectrum after inverse filtering. Higher-order filters capture more details of the formant structure, while lower-order filters are sometimes more appropriate because they maintain some signal characteristics without fully flattening them. The LPC model of the vocal tract effectively represents the resonant frequencies. By inverse filtering, these resonances are removed, leading to a pseudo-whisper [26]. Consider a sample speech signal. The spectrums before and after inverse filtering are depicted in Figure 3.

Inverse filtering using LPC effectively removes the vocal tract resonances, creating a flattened spectrum and transforming speech into a pseudo-whisper. This technique offers significant utility in various speech processing applications, including audio (speech) data augmentation.

The research presented in [24] analyzes the Long-Term Average Spectrum (LTAS) of both normal speech and whisper. It demonstrates, through the calculation of cepstral distance, that the spectra of speech and whisper exhibit greater similarity after inverse filtering. Consequently, the mismatch between normal speech and whisper is reduced.

5. Materials and Methods

5.1. Natural Speech Databases

In this study, two speech databases containing whisper utterances were employed. The first one is the Whi-Spe database exploited in full capacity [31]. Whisper utterances from this database were utilized for training and augmentation, and neutral utterances are used for inverse filtering (creating pseudo-whisper samples). Recordings from the Whi-Spe dataset consist of fifty isolated words spoken by ten speakers (5 female and 5 male). Each word was repeated 10 times, both in normal and whispered phonation. As a result, the overall size of the database encompasses 10,000 recordings, which is roughly equivalent to two hours of speech. The recordings were carried out under controlled laboratory conditions. The speech samples were digitized with a sampling rate of 22,050 Hz, 16 bits per sample, using the mono technique and the linear PCM wav format. More details regarding the vocabulary, segmentation procedure, and quality control of this database can be found in [31].

The second database, known as DBtest, was created in a real-world setting to evaluate the robustness of the ASR systems. Ten volunteer students (5 males and 5 females, as in Whi-Spe) uttered words (two times each) from the Whi-Spe lexicon in a whispered voice. The recordings were made using a laptop and an integrated microphone in various rooms, with ambient noise Sound Pressure Level (SPL) of approximately 35 dBA. The database is publicly available in the repository [32]. To ensure consistency, all recordings from the two databases were resampled to 16 kHz and 16 bits per sample. Furthermore, the audio files in both databases follow the same naming convention.

5.2. Augmented Speech Databases

Based on the Whi-Spe database comprising whisper and normal phonation, we created 22 artificial (augmented) datasets for experimental purposes. Augmented datasets were generated using MATLAB^® R2021b [33]. To examine the influence of three different augmentation techniques and their combinations, 14 augmented datasets were created. The flow diagram for the generation of augmented speech databases is depicted in Figure 4.

An additional 8 augmented datasets are generated for examination of the influence of the number of augmentations on the performance of the recognizer. The flow diagram for the generation of augmented speech databases using parallel augmentations is depicted in Figure 5.

5.3. Automatic Speech Recognition Systems

In this research study, the recognition performance using two types of classifiers was analyzed:

• Hidden Markov Models (HMM), and
• Convolutional Neural Networks (CNN).

The subsequent subsections will provide a more detailed explanation of both classifiers.

5.3.1. HMM-Based ASR System

The conventional ASR methodology relies on Hidden Markov Models (HMM) with Gaussian Mixture Models (GMM). Three modeling units for the recognition of isolated words are usually considered: phonemes independent from context (monophones), phonemes with contextual dependencies (biphones or triphones), and whole words. The highest robustness in whispered speech recognition and the Whi-Spe database is demonstrated using monophone models [34].

The most important configuration parameters in the feature extraction procedure are summarized in

Table 2. Configuration parameters in the feature extraction procedure.

Parameter	Name/Value
Feature vector	MFCC
Window	Hamming
Pre-emphasis	0.97
Window size/frame shift	24/8 ms
Cepstrum	Magnitude
Number of channels	26
Number of coefficients (static + ∆ + ∆∆)	39 (13 + 13 + 13)
Normalization	CMS

. The feature vectors were extracted using the HTK software (version 3.4.1) [35].

The developed ASR system utilized HMM models with continuous density GMMs and diagonal covariance matrices. Models of monophones consist of 5 states (3 emitting states) with a left-to-right topology and without skips, as depicted in Figure 6.

The models are initialized with the global mean and variance, with flat-start initialization. Baum-Welch re-estimation was performed for five cycles, and the number of mixtures was gradually increased up to 16. Phone-level transcription involved 32 monophones, encompassing 30 Serbian letters, the schwa phoneme, and silence. Although the phoneme schwa does not have a letter in Serbian, it is marked separately if it is next to the letter/r/. During the testing phase, the Viterbi decoder was employed to estimate the most probable state sequence. The ASR system was developed using the HTK toolkit, while MATLAB software (version R2016a) was utilized for generating scripts and configuration files, phonetic transcription, and logging the performance of the ASR system, which was evaluated within HTK.

5.3.2. CNN-Based ASR System

Even though Convolutional Neural Networks were originally designed for image recognition tasks, recent studies demonstrated promising performance in speech recognition, particularly for isolated and control words [36].

A typical CNN model architecture consists of the following components:

• Input layer,
• Convolution layer with an activation function,
• Pooling layer, and
• Fully Connected Layer

The most important parameters for the CNN model are summarized in

Table 3. The parameters of the CNN model.

Parameter	Name/Value
Number of convolution layers	2
Number of filters	32 and 64
Activation function	ReLU
Pooling layer	Max
Number of epochs	10
Optimizer	ADAM
Loss function	Categorical cross-entropy
Metric	Accuracy

.

In order to normalize input data (set between 0 and 1), the following transformation was applied:

$$norm\left(x\right)={\displaystyle \frac{x-x_{min}}{x_{max}-x_{min}}}$$

(6)

This is necessary because deep learning algorithms work more properly when the input data are normalized. Various algorithms are available for training neural networks, and the SGD algorithm and its variants are commonly used for training CNN models [37].

The speaker-independent mode was analyzed using 90% training and 10% validation split. The remaining speaker was reserved for testing employing a leave-one-speaker-out cross-validation approach. To ensure reliable evaluation of the performance, the experiments were repeated 10 times, and the average accuracy was calculated for each speaker.

After training both models, the training histories were obtained, in which training and validation losses were plotted against the epochs. Figure 7 shows the training history of the model. As shown, the training process was completed correctly, as the validation loss decreased along with the training loss among the epochs.

The speech signal is analyzed on a short-time basis, where it is divided into overlapping frames, and a feature vector is computed for each frame. The window size is typically between 20 and 30 ms, with consecutive frames shifted between 10 and 15 ms. Consequently, utterances of varying durations have different numbers of feature vectors. To mitigate this limitation, two approaches are considered:

• The window size is proportional to the frame period, $w_s=K{f}_p$, where K is a constant for all utterances (Figure 8).
• The window length is fixed, but K is dynamically changed.

Research studies, such as the one presented in [38], have shown better results in the recognition of isolated words using the first approach. The best results in recognizing words from the Whi-Spe database (in the speaker-dependent case) were achieved using 18 windows per utterance [39]. Therefore, in our experiments, segmentation into 18 overlapping windows and K = 3 was employed. Speech parameterization involved using 39 MFCC dynamic feature vectors per frame (13 static + Δ + ΔΔ), with cepstral mean subtraction (CMS) applied. Ultimately, each utterance was represented by a vector of 702 coefficients (18 × 39). The ASR recognizer was developed using Python software (version 3.9.12), utilizing the Scikit-learn package.

5.4. The Experiment Setup

The experimental setup flow chart is depicted in Figure 9 for the baseline experiments (a) and the experiments examining the impact of data augmentation techniques and the number of augmentations (b).

The accuracy is used to evaluate the performance of the recognizer. For instance, if N is the total number of utterances analyzed for a specific speaker and E is the number of incorrectly recognized utterances, the accuracy can be determined using the following equation:

$$Acc={\displaystyle \frac{N-E}{N}}·100\left[\%\right]$$

(7)

Word Error Rate (WER) is a common metric of the performance and can be calculated using Equation (8):

$$WER={\displaystyle \frac{E}{N}}·100\left[\%\right]$$

(8)

To ensure a more reliable evaluation of performance, a 10-fold cross-validation is conducted for every speaker. Finally, the average accuracy is calculated and employed as the final metric of the recognizer’s performance. The experiments were conducted on two laptops with similar specifications. The HMM training and testing were performed on a laptop equipped with an Intel Core i5-1235U processor, 1.30 GHz, and 8 GB of RAM. On the other hand, the laptop employed for CNN training and testing was equipped with an AMD Ryzen 5 4500U, 2.38 GHz, and 8 GB RAM.

6. Results

This section is structured in the following manner. Section 6.1 presents the results and discussion of the baseline experiments. The purpose of these experiments is to evaluate the recognition performance for two natural speech databases: Whi-Spe and DBtest, where the training is performed on the Whi-Spe database.

In Section 6.2, we investigate the impact of augmentation strategies on the ASR-recognizer (HMM and CNN) accuracy for the DBtest dataset.

In Section 6.3, we investigate the impact of the number of augmentations on the ASR-recognizer accuracy (HMM and CNN) for the DBtest dataset.

Section 6.4 delves into the examination of the cumulative impact of augmentation strategies and the number of augmentations on the enhancement of relative Word Error Rate (WER).

6.1. Baseline Experiments

The main objective of the baseline experiments was to analyze the speaker-independent recognition of a whisper in two distinct scenarios:

• Closed set—both training and testing were carried out using the Whi-Spe database.
• Open set—training was completed using the Whi-Spe database while testing was carried out with the DBtest database.

The training/test split for speaker-independent recognition is based on individual speakers, ensuring that utterances from the test speaker are not included in the training set. The accuracy results are presented in Figure 10 (HMM) and Figure 11 (CNN), showing the impact of varying percentages of the database used for the training, ranging from 10% to 100% with a 10% increment. The results of experiments for each speaker are given in the Supplementary Document accessible in the repository [32].

Figure 10 illustrates that the closed-set recognition for HMM achieved a final accuracy of 95.02% after full-capacity training. In contrast, the accuracy dropped significantly to 91.61% for the DBtest database. Bearing in mind that the DBtest is not recorded in a laboratory, a resulting drop in performance is expected. The most notable performance increase was observed between 10% and 20% of the training database, followed by a saturation effect and minor fluctuations after 30%.

The CNN recognition showed lower performance than HMM, as shown in Figure 11. The accuracy for full-capacity training was 92.46% (Whi-Spe) and 84.14% (DBtest). In contrast to HMM, the CNN scenario did not achieve performance saturation for the DBtest database. The insufficient size of the training database may have contributed to the poor performance of CNN.

The same experiment setup was performed for normal speech and Whi-Spe database. Compared to whispered speech, the closed set recognition of normal speech in baseline experiments was with higher success and reached an accuracy of 98.07% (HMM) and 92.80% (CNN) [40].

6.2. The Impact of Data Augmentation Strategies

Of the several augmentation techniques outlined in Section 3, three methods (single) and their combinations (2-way and 3-way) were examined in this study, resulting in a total of seven different augmentation techniques.

• Pitch Shift (PS),
• Time Stretch (TST),
• Volume Control (VC),
• PS + TST,
• TST + VC,
• PS + VC, and
• PS + TST + VC.

The probability of applying the augmentation method was set to 0.5 (see Figure 2a). The sequential augmentations had the following parameter ranges:

• PS—Semitone Shift Range: [−2, 2]
• TST—Speedup Factor Range: [0.8, 1.2]
• VC—Volume Gain Range: [−3 dB, 3 dB]

The values (real numbers) for the augmentation parameters are randomly assigned within these specified ranges. It is worth noting that adding noise and time shifting were excluded from the analysis due to their lack of positive influence on the performance of the ASR system.

The average word recognition accuracy for the speech database DBtest can be found in

Table 4. Average recognition accuracy (DBtest database) for seven data augmentation techniques and HMM framework. The accuracies with statistically significant improvements are bolded.

Augmentation
Training Scenario	PS	TST	VC	PS + TST	PS + VC	TST + VC	PS + TST + VC
W	90.66	90.66	91.61	89.06	90.75	90.98	89.88
P	72.16	71.32	69.51	72.11	72.06	72.26	70.84
OW	91.29	91.21	91.61	91.09	91.29	91.49	91.14
OP	91.17	90.91	91.34	91.06	91.20	91.09	90.88
OWP	91.53	91.91	92.09	91.37	91.48	92.06	91.52
ORIGINAL (NO AUG)	91.61

(HMM) and

Table 5. Average recognition accuracy (DBtest database) for seven data augmentation techniques and CNN framework. The accuracies with statistically significant improvements are bolded.

Augmentation
Training Scenario	PS	TST	VC	PS + TST	PS + VC	TST + VC	PS + TST + VC
W	84.29	83.72	83.88	84.22	84.37	83.70	84.93
P	69.52	69.76	69.23	69.94	69.00	69.72	68.85
OW	85.82	84.98	85.18	85.62	85.84	85.02	85.80
OP	85.73	85.80	86.17	86.10	85.74	85.16	85.80
OWP	87.54	87.37	87.28	87.97	88.06	86.86	87.31
ORIGINAL (NO AUG)	84.14

(CNN). The codes for experiments are accessible in [32]. In these experiments, the speech database used for augmentation is of the same size as the original one (sequential augmentation, as illustrated in Figure 2).

Moreover, the analysis included training solely with augmented speech samples (W—whisper augmented and P—pseudo-whisper augmented), as well as the fusion of original and augmented datasets (OW, OP, and OWP raws).

The results obtained from training using solely augmented utterances (W and P raw in Table 4 and Table 5) indicate the following:

• When augmentation was applied to whispered utterances (W raw), the performance of the HMM recognizer slightly weakened, while it remained comparable for the CNN recognizer.
• When augmentation was applied to pseudo-whisper utterances only (P raw), the performances significantly worsened for both the HMM and CNN recognizer.

When training was applied to both the original and augmented datasets (OW, OP, and OWP raw in Table 4 and Table 5), the performances were notably improved for the CNN recognizer for all techniques. At the same time, only a particular scenario and augmentation method led to performance improvement for the HMM recognizer. The significance of the inverse filtering algorithm is evident when examining the OWP and OW rows in Table 4 and Table 5. In every augmentation scenario, adding pseudo-whisper utterances in training has resulted in a marked improvement in accuracy for both recognizers.

In order to evaluate the statistical significance of obtained results in cases when augmented datasets are added to the original, a one-tailed Wilcoxon test for hypothesis H_A was performed.

• Hypothesis H_0A: “Both recognizers (initial recognizer when training was applied using only original samples and the other with augmentation—OW/OP/OWP) produce the same accuracy”.
• Hypothesis H_1A: “Recognizer with augmentation—OW/OP/OWP gives higher accuracy”.

The p-values for H_0A/H_1A hypothesis are provided in

Table 6. The p-value calculated for hypothesis H_A in relation to seven augmentation strategies (OW—original + whisper; OP—original + pseudo-whisper; OWP—original + whisper + pseudo-whisper). The statistically significant p-values are bolded.

Augmentation
Recognizer	Training Scenario	PS	TST	VC	PS + TST	PS + VC	TST + VC	PS + TST + VC
	OW	1.000	0.999	1.000	1.000	1.000	0.903	1.000
HMM	OP	0.998	1.000	0.985	0.999	0.998	1.000	1.000
	OWP	0.811	0.007	0.002	0.968	0.957	0.001	0.804
	OW	0.010	0.122	0.027	0.006	0.021	0.097	0.004
CNN	OP	0.018	0.017	0.002	0.016	0.007	0.049	0.018
	OWP	0.001	0.001	0.001	0.001	0.001	0.002	0.001

for values of alpha 0.05.

As can be seen from Table 6, statistically significant improvement ($p<0.05$) in the accuracy for the HMM recognizer is obtained only in the OWP case for TST, VC, and TST + VC augmentation strategies. Statistically significant improvement is obtained for all techniques in the OWP and OP cases for the CNN recognizer.

Processing time is a crucial metric for evaluating the efficiency during the training phase and real-time processing capabilities in the decoding phase of ASR systems. The Real-Time Factor (RTF) measures the speed of ASR during decoding, defined as the ratio between the processing time and the input duration. Figure 12 shows the training time and RTF for the best augmentation strategies (VC for HMM and PS + VC for CNN).

The results obtained suggest the following conclusions:

• Expanding the training dataset by incorporating augmented utterances significantly increases the training time but improves accuracy, particularly for the CNN recognizer.
• The RTF for the HMM recognizer is considerably higher and remains relatively stable, even with the expansion of the training dataset.

6.3. The Impact of the Number of Augmentations

As Figure 2b in Section 3 shows, parallel augmentation enables the enlargement of augmented dataset size. By defining the number of augmentations (N), we obtain an N-times larger augmented dataset. The objective of this subsection is to examine the impact of N on the performances of the recognizers. Because of a larger training corpus, the number of mixtures for the HMM recognizer is increased from 16 to 32.

Figure 13 depicts the average accuracy of HMM and CNN recognizers for the number of augmentations from one to five. The best augmentation strategy from the previous subsection (VC for HMM and PS + VC for CNN in the OWP scenario) was exploited. The results of particular experiments for each speaker are given in the Supplementary Document accessible in the repository [32].

As evident from Figure 13, unlike the CNN, the HMM recognizer shows a small drop in accuracy after N = 2. In that way, enlargement of the training part does not lead to performance improvement for the HMM recognizer.

On the other hand, the accuracy for the CNN recognizer notably increases as N goes higher. When N goes from 1 to 3, the accuracy roughly increases by 1 percentage point (88.06 to 89.03). Afterward, the improvement slowly decreases with a tendency for saturation of performance. The final examined case (N = 5) gives an accuracy of 89.45%. The next subsection deals with the relative WER improvement.

6.4. WER Improvement

The relative WER improvement (i.e., WER reduction) with respect to the cumulative effect of augmentation techniques (VC for HMM and PS + VC for CNN), inverse filtering, and parallel augmentations is depicted in Figure 14. As can be seen, the contribution of augmentation techniques and inverse filtering to the WER improvement is 24.7% for CNN and 5.7% for HMM. Further increment in the number of augmentations leads to the WER improvement for the CNN recognizer and reaches 33.5% for N = 5.

For the HMM recognizer, the WER is improved only in the case when N is changed from 1 to 2. Further increment leads to the WER deterioration.

It should be emphasized that adding pseudo-whisper samples obtained through inverse filtering from normal speech contributes significantly to the WER improvement, especially for the CNN recognizer.

It is crucial to bear in mind that the accuracy of ASR systems is highly dependent on factors such as the size of the speech database employed in training (including the number of speakers and utterances), perplexity, and the type of speech (spontaneous, conversational, read, isolated-word, etc.). Therefore, conducting direct and consistent comparisons between different studies may not always be feasible.

To the best of our knowledge, there are only a few research studies that report WER improvement through the utilization of data augmentation and pseudo-whisper generation methods to enhance speaker-independent whisper recognition.

The results of this research demonstrate a greater relative improvement compared to the state-of-the-art studies carried out on the wTIMIT speech corpus [41], outperforming the improvements reported in [15] (23%) and [18] (18%). However, to enable a direct comparison, it is important to assess our methodology on the wTIMIT corpus, which can be explored in future studies.

7. Conclusions

The motivation behind the research study presented in this paper is the increasing necessity to enhance human-machine speech communication, including variations in vocal effort. Given the substantial acoustic mismatch between normal and whispered speech, achieving reliable and accurate speaker-independent recognition of whispered speech in real-world scenarios poses a significant challenge for researchers.

Recent research studies have demonstrated the viability of utilizing data augmentation strategies to enhance the robustness of ASR systems, particularly those utilizing deep learning algorithms. The efficacy of traditional augmentation techniques such as pitch shifting, time stretching, volume control, and their combinations is assessed. Furthermore, the potential of generating pseudo-whispers through inverse filtering algorithms is explored.

The findings of this study suggest that data augmentation can be successfully utilized to enhance the performance of isolated-word whispered speech recognition systems. The conducted experiments have shown that the success of the recognizer, besides traditional augmentation methods, can be additionally improved by an inverse filtering strategy for both HMM and CNN recognizers. Conducted statistical tests confirmed the significance of improvement in performance.

The direction of future works includes enlarging the dataset of whispered speech used for training along with testing evaluated algorithms on different whisper datasets that are currently available.