Enhancing Far-Field Speech Recognition with Mixer: A Novel Data Augmentation Approach

Abstract

Recent advancements in end-to-end (E2E) modeling have notably improved automatic speech recognition (ASR) systems; however, far-field speech recognition (FSR) remains challenging due to signal degradation from factors such as low signal-to-noise ratio, reverberation, and interfering sounds. This requires richer training data and multi-channel speech enhancement. To address this gap, we introduce Mixer, a novel data augmentation technique designed to further enhance the performance of large-scale pre-trained models for FSR. Mixer interpolates and mixes feature representations of speech samples and their corresponding losses, extending the MixSpeech framework to intermediate layers of Whisper. Additionally, we propose Mixer-C, which further leverages multi-channel information by combining speech from different microphone channels using a channel selector. Experimental results demonstrate that Mixer significantly outperforms existing methods, including SpecAugment, achieving a relative word error rate (WER) reduction of 3.6% compared to the baseline. Furthermore, Mixer-C offers an additional WER improvement of 2.2%, showcasing its efficacy in improving FSR accuracy.

1. Introduction

In recent years, end-to-end (E2E) modeling techniques have significantly improved the accuracy of automatic speech recognition (ASR) systems [1,2,3]. However, challenges persist in far-field speech recognition (FSR) due to signal degradation. Firstly, speech signals attenuate as they propagate over distance, leading to a low signal-to-noise ratio (SNR). Secondly, in enclosed environments such as living rooms or conference rooms, the source signal undergoes multiple reflections from walls and indoor objects, resulting in multipath propagation and a temporal ambiguity phenomenon known as reverberation. Thirdly, microphones may capture unwanted interfering sounds which are often diverse and non-stationary, making them difficult to filter out. All these factors negatively impact the performance of FSR.

Given the signal degradation issues in far-field speech recognition, the processing pipeline diverges significantly from ASR. Firstly, the model is trained on data representative of the typical signal degradation encountered in far-field scenarios, necessitating richer training data. Secondly, microphone arrays are predominantly used instead of single microphones to capture sound, which enables multi-channel speech enhancement, proven highly successful in noisy reverberant environments.

Data augmentation (DA) strategies are crucial as they can generate virtually unlimited training samples [4]. Notably, SpecAugment (SA) [5] has demonstrated significant improvements in the accuracy and noise robustness of end-to-end (E2E) ASR systems. However, to our knowledge, relatively few studies have applied data augmentation techniques to multi-channel FSR. Existing attempts primarily include speed/volume perturbation [6], white noise injection [7], and channel dropout [8], indicating further innovation potential in this area. In response to this gap, we propose Mixer, a method that effectively enhances the performance of large-scale pre-trained models for far-field speech recognition. Mixer achieves this by interpolating and mixing feature representations from speech samples and their corresponding losses. According to the MixSpeech paper given in [9], it only mixes the input speech, while Mixer is actually a generalization of MixSpeech. The Mixer method can not only mix the input speech but also mix the intermediate layer features output via the Whisper encoder, thereby offering a more robust framework for FSR.

Furthermore, since far-field speech recognition employs multiple microphone arrays to capture sound, single-channel data may be incomplete due to factors such as speaker position and movement. To enhance multi-channel speech, we further propose Mixer-C, which selects and combines speech from different channels with the same label through a channel selector. This approach strengthens the multi-channel information of FSR, enabling a more comprehensive understanding for speech and thereby enhancing performance. Additionally, by mixing different channels of speech, Mixer-C addresses a significant limitation that relies on the same array configuration during both training and testing, improving the model’s adaptation to various array configurations.

Although large-scale pre-trained models have demonstrated exceptional performance across various tasks, the immense parameter size of these models makes fine-tuning computationally expensive and prone to overfitting. Therefore, we employ the LoRA [10] algorithm to investigate the performance of FSR on Whisper as our baseline. Our main contributions are as follows: (a) We explore efficient fine-tuning methods based on Whisper for far-field speech recognition (FSR). (b) We introduce a data augmentation method, Mixer, specifically designed for FSR, which significantly enhances model performance. (c) We present Mixer-C, an extension of Mixer, to further leverage multi-channel information for improved accuracy. (d) Experimental results demonstrate that Mixer consistently outperforms existing methods such as SpecAugment across all versions of Whisper, achieving a relative word error rate (WER) reduction of 3.6% compared to the baseline. Moreover, Mixer-C further enhances the performance of Mixer, yielding an additional WER improvement of 2.2%.

2. Related Work

2.1. Far-Field ASR

Recently, large-scale pre-trained models have demonstrated notable success in FSR [11]. However, far-field data often suffer from signal degradation issues, prompting the development of specialized data augmentation strategies. Channel dropout [12] was introduced as a method for dropping frequency bands within a single-channel signal, similar to SpecAugment. To enhance multi-channel information, Shimada et al. [13] utilized audio channel dropping to augment data for training a speaker verification system that leverages beamforming as its frontend. Moreover, ChannelAugment [8] was proposed to address the limitation of both training and testing data being derived from a fixed array geometry. The distinguishing aspect of our research is the integration of different channels from multi-channel inputs during end-to-end (E2E) ASR training. To the best of our knowledge, this approach has not been explored in previous studies.

2.2. Mixing Methods

Data augmentation is a crucial component for successful deep learning. It helps mitigate sample size issues and prevents overfitting, effectively reducing the epistemic uncertainty of the model [14]. Mixup [15] is a simple yet powerful data augmentation technique that improves the performance and robustness of many state-of-the-art models. Given its effectiveness, Mixup has been extended into various derivatives. Manifold Mixup [16] combines input features within the network’s hidden layers. Additionally, Cutmix [17] proposes mixing patches to create more visually meaningful augmentations. Patchup [18] extends this idea by mixing patches in the hidden space. Progress has also been made in applying these techniques to natural language processing. In their research, Guo et al. [19] investigated the application of Mixup for sequence classification tasks. Their findings suggest that blending words and sentences can produce advantageous outcomes. Along similar lines, Ssmix [20] was proposed, featuring a method where less critical words are replaced based on saliency, thereby enhancing the overall performance. MixText [21] applied Mixup for semi-supervised text classification, whereas MixSpeech [9] adapted it for the speech domain. Moreover, MixLoss [22] generalized MixSpeech to sequence generation tasks.

3. Methods

3.1. Model Structure

Whisper [2] is a transformer-based encoder–decoder model for handling various speech tasks, including speech recognition, speech translation, language identification, etc. Assuming the input log-Mel spectrogram feature vectors denoted as $X=\{x_1,x_2,\dots ,x_T\}$, and a target transcription is represented by a sequence of N tokens denoted as $y=\{y_1,y_2,\dots ,y_N\}$. The input X is initially processed by two 1-dimensional convolution layers, and is then encoded to the hidden vector H by the encoder of Whisper. Next, the decoder consumes H, a prompt S, and the previously generated tokens $y_{<j}$ as inputs to predict the next token $y_j$. The whole process can be described as follows:

$$\left\{\begin{array}{c}H=Encoder\left(Conv\right(X\left)\right)\hfill \\ y_j=Decoder(y_{<j},H,S)\hfill \end{array}\right.$$

(1)

where the prompt S used can be denoted as $[\langle |\mathrm{sop}|\rangle ,\mathrm{previous}\mathrm{text},\langle |\mathrm{sot}|\rangle ,\langle |\mathrm{language}|\rangle ,$$\langle \left|\mathrm{task}\right|\rangle ,\langle \left|\mathrm{no}\mathrm{times-tamps}\right|\rangle ].$ The parts encapsulated within $\langle |·|\rangle$ are special tokens. To specify the FSR task, we set the $\langle \left|\mathrm{task}\right|\rangle$ to “transcribe”, and $\langle \left|\mathrm{language}\right|\rangle$ is set as “en”. The “previous text” is an optional sequence of text tokens for decoding control. However, directly adapting such large models to low-resource scenarios can be challenging due to overfitting issues and high computational costs. Therefore, parameter-efficient fine-tuning [10,23,24] has been proposed, which involves freezing the pre-trained model and training specific small additive modules, such as Adapter [24], LoRA [10] and reprogramming [25,26,27].

LoRA [10], introduced in NLP, is designed to adapt large language models (LLMs) for specific domains or tasks more efficiently. Researchers have noted that pre-trained LLM weights occupy a low-dimensional intrinsic space. Leveraging this insight, LoRA keeps the original weights unchanged while updating only the low-rank incremental weight matrices. For instance, consider the i-th linear projection in the forward pass, represented as $f_i\left(X\right)={W_i}^{T}X+b_i$, where $W_i\in {\mathbb{R}}^{d1\times d2}$ and $b_i\in {\mathbb{R}}^{d2}$ are the frozen weight and bias. LoRA modifies the forward computation to include updates to only these low-rank matrices $\Delta W_i$, which can be expressed as follows:

$$f_i\left(X\right)={(W_i+\Delta W_i)}^{T}X+b_i$$

(2)

In this equation, $\Delta W_i$ is calculated as the product of two rank-decomposed matrices, $B_i$ and $A_i$, such that $\Delta W_i=B_i{A}_i$. The matrices $B_i\in {\mathbb{R}}^{d1\times r}$ and $A_i\in {\mathbb{R}}^{r\times d2}$ are the two trainable matrices, with rank $r\ll min\{d1,d2\}$. When deployed in production, we can explicitly compute and store the combined weight matrix W as $W=W_i+B_i{A}_i$, and proceed with inference as normal. Consequently, LoRA is designed to introduce no additional latency during inference compared to a fully fine-tuned model. Therefore, we employ the LoRA algorithm for Whisper as our baseline, with a detailed illustration shown in Figure 1.

3.2. Mixing Methods

Mixup [15] generates new augmented data through a process of linear interpolation, as described in the following equations:

$$\left\{\begin{array}{c}\widehat{x}=\lambda x_i+(1-\lambda )x_j\hfill \\ \widehat{y}=\lambda y_i+(1-\lambda )y_j\hfill \end{array}\right.$$

(3)

where $(x_i,y_i)$ and $(x_j,y_j)$ represent two randomly sampled examples from the training data. The mixed input feature $\widehat{x}$ and the corresponding mixed label $\widehat{y}$ are generated by blending the i-th and j-th examples. The mixing weight $\lambda \sim Beta(\alpha ,\alpha )$ with $\alpha >0$. When $\alpha =1.0$, $\lambda$ is uniformly sampled from the interval [0,1], which also means $\lambda \in U(0,1)$. By leveraging the prior knowledge that linear interpolations of inputs yield linear interpolations of the corresponding labels, Mixup effectively expands the training distribution. Moreover, some studies [28,29] have theoretically demonstrated that Mixup constitutes a form of data-adaptive regularization, which proves to be highly effective in improving the model’s generalization capabilities.

Manifold Mixup [16] is an extension of Mixup [15], allowing for the linear interpolation of representations from any layer within a neural network. Consider a neural network $f_k\left(g_{\mathrm{k}}\right)$, where $g_k$ maps the input data into the hidden representation at k-th layer, and $f_k$ subsequently maps the hidden representation to the final output $f\left(x\right)$. For a randomly selected k-th layer, we process two distinct mini-batches of random data $(x_1,y_1)$ and $(x_2,y_2)$ as usual, until we reach the k-th layer. This provides us with two intermediate mini-batches $(g_{\mathrm{k}}\left(x_1\right),y_1)$ and $(g_{\mathrm{k}}\left(x_2\right),y_2)$. Subsequently, we construct a mixed mini-batch as follows:

$$({\tilde{g}}_k,\tilde{y}):=(Mi{x}_{\lambda }(g_k\left(x_1\right),g_k\left(x_2\right)),Mi{x}_{\lambda }(y_1,y_2))$$

(4)

where $Mi{x}_{\lambda }(a,b)=\lambda a+(1-\lambda )b$. The mixing coefficient $\lambda \sim Beta(\alpha ,\alpha )$ with $\alpha >0$. This interpolation yields a novel training example based on the hidden representation of the model, effectively serving as a powerful data augmentation technique. Notably, Mixup becomes a special case of Manifold Mixup [16] when k is set to 0.

Mixup is inherently suited for classification tasks but encounters challenges when applied to speech recognition due to several fundamental reasons: firstly, text sequences $(Y_i,Y_j)$ often exhibit varying lengths, making direct mixing impractical; secondly, the discrete nature of text sequences prohibits straightforward addition; thirdly, mixing two text sequences in the embedding space may lead the model to predict a mixture embedding, which could confuse the ASR model and potentially degrade performance, as the ultimate goal is to recognize a single text transcript from the speech input. Therefore, Meng et al. [9] introduced MixSpeech for speech recognition, which blends two input sequences and applies two separate loss functions in the following manner:

$$\left\{\begin{array}{c}\widehat{X}=\lambda X_i+(1-\lambda )X_j\hfill \\ L_i=L(\widehat{X},Y_i),L_j=L(\widehat{X},Y_j)\hfill \\ \widehat{L}=\lambda L_i+(1-\lambda )L_j\hfill \end{array}\right.$$

(5)

where $X_i$ and $Y_i$ represent the input speech and corresponding label for the i-th sample, respectively. $\widehat{X}$ denotes the mixed speech obtained by a frame-wise combination of two speech sequences with mixing factor $\lambda$. L denotes the ASR loss, and $\widehat{L}$ denotes the mixed loss. Following Mixup, $\lambda \sim Beta(\alpha ,\alpha )$ with $\alpha >0$.

3.3. Mixer of FSR

In far-field speech recognition (FSR) scenarios, speech signals attenuate as they propagate over distance, resulting in a decreased signal-to-noise ratio (SNR). Additionally, these signals undergo reflections from surfaces and diffraction around obstacles, leading to reverberation. Moreover, microphones frequently capture unwanted environmental noise. These effects distort the speech signal in different manners, and thereby require much more training data to achieve high accuracy and generalization.

We propose a data augmentation method for far-field ASR, called Mixer, which mixes any representations of an E2E ASR model. Consider the input log-Mel spectrogram feature vectors denoted as $X=\{x_1,x_2,\dots ,x_T\}$, and that a target transcription is represented by a sequence of N tokens denoted as $Y=\{y_1,y_2,\dots ,y_N\}$. For an arbitrary K-layers model like Whisper, we denote $g_k(·)$ as the function that processes data from the input to the k-th layer, and $f_k(·)$ as the function that maps data from the k-th layer to the output. As is seen in Figure 2, Mixer includes two stages to implement. First, Mixer mixes any two feature representations $g_k\left(X_i\right)$ and $g_k\left(X_j\right)$ extracted from audio samples:

$$g_k\left(\widehat{X}\right)=\lambda g_k\left(X_i\right)+(1-\lambda )g_k\left(X_j\right)$$

(6)

where $g_k\left(X_i\right)$ denotes the output of k-th layer and k is the layer of the model that requires mixing. Second, Mixer trains the model to optimize the mixed loss functions by aligning the mixed weight to the corresponding loss. The process can be denoted as follows:

$$L_{\mathrm{mix}}=\lambda L(f_k\left(g_k\left(\widehat{X}\right)\right),Y_i)+(1-\lambda )L(f_k\left(g_k\left(\widehat{X}\right)\right),Y_j)$$

(7)

where $\lambda \sim ϵBeta(\alpha ,\alpha )$ with $\alpha >0$. And $ϵ$ is the maximum value of the mixing weight. By aligning the interpolation weight in the mixing of features with that of the loss, the method establishes a linear relationship between the input and output space of the model. When k = 0, since the hidden representations are log-Mel spectrograms of the input, Mixer naturally extends the MixSpeech [9] method.

3.4. Mixer-C of FSR

We further propose a simpler mixing method tailored for far-field ASR, called Mixer-C. Mixer-C interpolates speech samples from the different channels with the same labels to augment the channel information. In far-field speech recognition, there exist multi-channel data collected by multiple microphones. The multi-channel information is instrumental in enabling models to capture not only the spectral characteristics of the target and interference signals but also their spatial attributes, as recorded by the different microphones. This added dimension is highly advantageous for improving the effectiveness of far-field automatic speech recognition (ASR). The multi-channel speech dataset can be denoted as $D=\{D_1,D_2,\dots ,D_C\}$, where C signifies the numbers of channels. Each speech sample ${X_i}^c$ from the c-th channel is associated with the same label $Y_i$ as its counterparts in all other channels, denoted as ${X_i}^{-c}$, where $-c$ signifies all channels except for c, i.e., $-c=\{1,\dots ,c-1,c+1,\dots ,C\}$. As is shown in Figure 2, Mixer-C includes two steps: first, the channel selector selects the paired speech data ${X_i}^m$ and ${X_i}^n$ from different channels with the same label from the audio samples, and then mixes the selected speech data in the k-th layer of the neural network, which can be formulated as

$$\widehat{X}=\lambda g_k\left({X_i}^m\right)+(1-\lambda )g_k\left({X_i}^n\right),m\ne n$$

(8)

where ${X_i}^m\in D_m$ and ${X_i}^n\in D_n$ correspond to speech samples from different channels that share the same label $Y_i$. Due to the two samples sharing the same label, Mixer-C does not require mixing the loss. Mixer-C can enhance the model’s understanding of information from different channels. Compared to Mixer, this approach does not require any additional computation of the loss function, thus improving computational efficiency.

4. Experimental Setup

4.1. Datasets

The DiPCo [30] dataset is an open speech dataset, designed to simulate a family dinner party scenario, addressing the challenges of speech separation and recognition in multi-speaker environments. The dataset was recorded by Amazon employee volunteers engaged in natural conversations within everyday household settings. Specifically, it includes distant speech captured by 5 far-field devices, each equipped with a 7-mic circular array, totaling 35 microphones. As the DiPCo dataset is one of the benchmark datasets in the CHiME-challenge series, it i used to evaluate far-field speech recognition performance across multiple scenarios [11,31].

The dataset was initially preprocessed using the Weighted Prediction Error (WPE) [32] method to enhance the signals for each utterance. After WPE processing, the multi-channel audio from each array was transformed into single-channel audio using the array-based BeamformIt [33] algorithm. Consequently, the multi-channel audio from multiple array devices was consolidated into a single multi-channel audio stream, with the number of channels corresponding to the number of far-field devices.

We utilized the development set for training and evaluation, and the test set for testing purposes. Subsequently, we filtered the audio and label lengths, excluding any audio segments shorter than 3 s or longer than 30 s. For the text content, we filtered out transcripts with fewer than 3 words or more than 100 words.

4.2. Experimental Details

We utilized the ESPnet [34] toolkit for data processing, resulting in enhanced channel audio. Our experiments were based on the Whisper model, including several different versions: Whisper-Small (244 M), Whisper-Medium (768 M), Whisper-Large-v2 (1550 M), and Whisper-Large-v3 (1550 M), all with English as the prompt language. Given the immense size of our dataset, fine-tuning every model comprehensively poses significant computational challenges. Hence, we adopted an efficient fine-tuning strategy, using the LoRA method to fine-tune some parameters. For LoRA, we set r = 4, which resulted in 2.21 M training parameters for the large version of model. We fine-tuned the model for 5 epochs using a batch size of 8, utilizing the Adam optimizer with a learning rate of 1 $\times {10}^{-3}$ and Word Error Rate (WER) as the evaluation metric. The default $\alpha$ for the Beta distribution was set to 2, and the default mixing maximum value $ϵ$ was set to 1. The proportion of one batch data to be trained on using Mixer is denoted as $\tau$, which was set to 15%. And we set the default mixed layer k to 0.

5. Results

5.1. Main Results

We compare Mixer with four different configurations: (1) direct testing performance of the pre-trained model; (2) models fine-tuned using the DiPCo dataset; (3) models trained using LoRA without Mixer (Baseline); and (4) models trained with SpecAugment, an effective data augmentation technique. Given our limited computational resources, we did not proceed with fine-tuning the entire Whisper-Large-v2 or Whisper-Large-v3 models. The experiment results are shown in

Table 1. WER (%) of four different Whisper versions under different training settings. “Params” denotes the trainable parameters of LoRA.

	Whisper-	Whisper-	Whisper-	Whisper-
	Small	Medium	Large-v2	Large-v3	Avg.
Params	0.4977 M	1.1327 M	2.2118 M	2.2118 M	/
Testing	46.98	42.76	46.11	42.04	44.47
Finetune	59.07	49.36	/	/	/
LoRA	52.65	39.70	42.24	37.41	43.00
+SpecAugment [5]	62.94	39.14	40.56	41.84	46.12
+Mixer	49.15	38.46	38.50	36.07	40.55
+Mixer-C	51.29	39.82	40.29	35.27	41.66

. Firstly, it can be seen that the direct testing performance of Whisper is already commendable. Surprisingly, both the Whisper-Medium and Whisper-Small achieved comparable results to the Whisper-Large-v2. This is likely attributed to the presence of the DiPCo dataset in the training corpus of Whisper. Furthermore, owing to the vast capacity of the large model, Whisper-Large-v2’s performance on the DiPCo dataset actually declined, whereas Whisper-Large-v2 achieved the best results. This underscores that a combination of large model capacity and improved training still yields exceptional outcomes. Secondly, it can be seen that fine-tuning the full model actually reduced Whisper-Small and Whisper-Medium’s performances on the test set. This was primarily due to the substantial distribution discrepancy between the training and test data, causing the model to forget the original DiPCo data while overfitting to the current batch of training data. Thus, the limitations of fine-tuning the entire model become apparent.

Therefore, we employ the LoRA method for training, and for large models, it only requires fine-tuning of 2.21 M parameters. Initially, we observed that despite the Whisper-Small model’s underwhelming performance, it manages to mitigate the overfitting issue to some degree. Compared with other models, none exhibited overt overfitting, suggesting that the pre-trained initialization of Whisper-Small possesses limited capacity. For Whisper-Medium, utilizing efficient fine-tuning strategies effectively curbs the overfitting that can occur with full model fine-tuning, enabling effective learning. Furthermore, the Whisper-Large-v3 model surpasses the Whisper-Medium model, demonstrating its ability to acquire more knowledge and perform better, demonstrating the Whisper-Large-v3 model’s superiority.

Moreover, we compared the performance of the SpecAugment and Mixer data augmentation methods. Since the speech domain is different from images, there are few mixing-based data augmentation methods like CutMix [17] other than MixSpeech. Therefore, we compared our method with SpecAugment, a very classical data augmentation method in speech. We found that SpecAugment only achieved some performance improvements on the medium and Whisper-Large-v2 versions, but had no effect on Whisper-Small and Whisper-Large-v3. This indicates that it is difficult for SpecAugment to be effective when it comes to far-field speech recognition data with high noise and complex channel conditions. However, Mixer achieves performance improvements on all models and can achieve greater performance gains compared to SpecAugment. We analyzed that this is mainly because SpecAugment injects noise into the data through time-domain or frequency-domain masking to enhance the data, while the channel environment for far-field speech recognition is already very poor, and injecting noise will affect the model’s learning of the data. In contrast, Mixer fits the data distribution in a high-dimensional space through input and output interpolation, thereby enhancing the model’s learning.

For the Mixer-C method, it can be observed that it is effective on most versions of the models. It performs well on the Whipser-Large-v3 model, surpassing the performance of Mixer. However, it performs worse than Mixer on other versions of the models. We analyze this is closely related to the fundamental capabilities of the baseline model. The enhancement of multi-channel information is more effective in models with strong baseline capabilities and a larger model capacity; otherwise, it may confuse the model’s learning. However, Mixer is also superior to SpecAugment and does not overly confuse the model’s learning.

5.2. Method Analysis

The Effect of Different Mixing Proportions. We conducted an analysis on the influence of varying mixing proportions $\tau$ within Mixer, specifically examining the shifts in model performance across distinct mixing proportion of 15%, 30%, 50%, 80%, and 100%. The results are shown in Figure 3. Our findings indicate that, for the Whisper-Large-v2 model, achieving superior performance necessitates a higher proportion. It can be found that for the Whisper-Large-v2 model, a higher proportion of data needs to be mixed to achieve better performance. Although it has a considerable capacity, the Whisper-Large-v2 model is not well versed in this specific data type, as evidenced by its poor direct testing performance. Therefore, incorporating additional information about the data distribution can further enhance its performance. Conversely, in the case of the Whisper-Medium, its limited capacity coupled with an already strong baseline performance means that excessive enhancement of the data can inadvertently hinder the model’s ability to learn from genuine data to some extent. In contrast, the Whisper-Large-v3 model exhibits minimal sensitivity to variations in mixing ratios, attributed to its combination of high capacity and extensive knowledge of the data, which allows it to consistently deliver superior performance. But overall, it can be observed that the model is not very sensitive to the mixing proportion $\tau$, demonstrating the superiority of Mixer. Since the analysis in Table 1 shows that the Mixer-C is more suitable for the Whisper-Large-v3 model, we only analyzed the impact of different mixing proportions on Whisper-Large-v3. For Mixer-C, it can be found that mixing more data leads to better performance. This is understandable because for models with strong baseline performance, enhancing channel information allows the model to gain a more comprehensive understanding of the speech signal, thereby improving performance.

The Effect of Different Sampling Distributions. Since the $\lambda$ value is sampled from the Beta distribution, to explore the impact of different sampling distributions on the Mixer and Mixer-C, we analyzed the coefficient $\alpha$ of the Beta distribution.

Table 2. WER (%) of Mixer/Mixer-C under different hyperparameter $\alpha$.

$\mathit{\alpha }$	0.2	0.5	1	2	5	8
Mixer	39.50	37.45	37.93	36.07	37.61	38.94
Mixer-C	35.72	36.77	36.20	35.27	37.43	36.82

demonstrates the performance impact of the Mixer and Mixer-C on the Whisper-Large-v3 model under various $\alpha$ values. For Mixer, it can be observed that an $\alpha$ value of 2 achieves the best performance. Neither excessively large nor excessively small values of $\alpha$ yield satisfactory results. However, for Mixer-C, both an $\alpha$ of 2 and 0.2 yield good results. Overall, the choice of $\alpha$ has a relatively small impact on Mixer-C, indicating that Mixer-C is less sensitive to the hyperparameter $\alpha$.

The Effect of Maximum Mixing Values. Since we introduced $ϵ$ to adjust the maximum value of $\lambda$ sampled from the Beta distribution, we conducted an analysis on how the performance of the Mixer/Mixer-C varies under different maximum mixing values, as shown in Figure 4. Each data point in Figure 4 is accompanied by error bars, which are used to illustrate the uncertainty of the experimental results. The error bars come from experiments conducted on the same maximum mixing value at different mixing proportions (15%, 50%, 80%), reflecting the impact of different mixing ratios on the result.

For Mixer, it can be observed that when the maximum mixing value is 0.1, the model’s performance is better. As the maximum mixing value increases, Mixer’s performance begins to decline, reaching its lowest point at 0.5. When the ratio exceeds 0.5, the performance starts to rise, and the WER reaches its minimum value when the maximum mixing value of WER is 1. We have analyzed the reasons behind this phenomenon. Since $\widehat{X}=\lambda X_1+(1-\lambda )X_2$, a decrease in $\lambda$ leads to an increase in $1-\lambda$, and $\widehat{X}$ transitions from resembling $x_1$ more to resembling $x_2$ more. Consequently, when the maximum mixing value approaches 0 or 1, the change in $\widehat{X}$ relative to the input is actually smaller. However, when the maximum mixing value is close to 0.5, the change in $\widehat{X}$ is significant, which may introduce excessive fluctuations and affect the performance. Finally, we can observe similar experimental conclusions for Mixer-C, but it can be seen that the performance of Mixer-C is overall better than that of Mixer. And the performance is superior to the maximum mixing weight until 1. This indicates that enhancing channel information is very important for Whisper-Large-v3.

The Effect of Mixing Encoder Layer $\mathit{k}$. For large-scale pre-trained models, the entire encoder is devoted to feature extraction. Therefore, we experiment with mixing the features from different layers k of the encoder on Mixer and Mixer-C, and the final experimental results are shown in Figure 5 and Figure 6. When k is 0, it indicates that the input spectrum is being mixed. It can be observed that the results of mixing intermediate layers are not as good as those of mixing at the input, which may be due to the fact that the feature representations in intermediate layers are focused on different aspects, which confuses the model’s learning. However, mixing deeper layer features can also achieve good results, because the later layers, compared to intermediate layers, can encode more acoustic information. Moreover, the deeper features are also closer to the model’s decoding output, which can strengthen the connection between the linear enhancement relationships. Compared to Mixer, Mixer-C shows a significant fluctuation in performance across different layers, indicating that the mixing of channel information is more influenced by the form of feature representation. If the feature is not well suited for representing channel information, it may greatly affect the performance of Mixer-C.

Moreover, we also extend our methods to Manifold Mixer/Mixer-C as we did for Manifold Mixup [16], which is denoted as the dashline. It is clear that although Manifold Mixer/Mixer-C is effective, it is relatively limited and not as effective as directly mixing at the input layer. Compared to Mixer, Mixer-C generalizes better for manifold learning.

6. Conclusions

In this paper, we first propose a data augmentation algorithm called Mixer that mixes the feature representations and corresponding loss of Whisper. Mixer enhances the prior information deficiency caused by signal degradation in far-field speech recognition. Secondly, we further introduce Mixer-C, which mixes the feature representations from different channels with the same labels, enhancing the multi-channel information of far-field data. Extensive experiments demonstrate that both methods can effectively improve the performance of far-field speech recognition, and we analyze the characteristics and applicable scenarios of the two methods. The proposed methods are not only applicable to the Whisper model but can also be extended to other large-scale pre-trained models. This provides researchers with a general framework for optimizing and improving existing speech recognition systems, enabling them to perform better in various complex environments. The proposed method has broad application prospects in fields such as smart homes and in-vehicle voice systems. In the future, we will study more effective multi-channel information enhancement techniques in front-end processing.