Speaker-Attributed Training for Multi-Speaker Speech Recognition Using Multi-Stage Encoders and Attention-Weighted Speaker Embedding

Featured Application

Speech recognition; speaker adaptation; speaker diarization.

Abstract

Automatic speech recognition (ASR) aims at understanding naturally spoken human speech to be used as text inputs to machines. In multi-speaker environments, where multiple speakers are talking simultaneously with a large amount of overlap, a significant performance degradation may occur with conventional ASR systems if they are trained by recordings of single talkers. This paper proposes a multi-speaker ASR method that incorporates speaker embedding information as an additional input. The embedding information for each of the speakers in the training set was extracted as numeric vectors, and all of the embedding vectors were stacked to construct a total speaker profile matrix. The speaker profile matrix from the training dataset enables finding embedding vectors that are close to the speakers of the input recordings in the test conditions, and it helps to recognize the individual speakers’ voices mixed in the input. Furthermore, the proposed method efficiently reuses the decoder from the existing speaker-independent ASR model, eliminating the need for retraining the entire system. Various speaker embedding methods such as i-vector, d-vector, and x-vector were adopted, and the experimental results show 0.33% and 0.95% absolute (3.9% and 11.5% relative) improvements without and with the speaker profile in the word error rate (WER).

1. Introduction

Automatic speech recognition (ASR) aims at understanding naturally spoken human speech to be used as text inputs to machines so that they can understand the human speaker’s intent and further communicate with the talker. The natural speaking environments include speaking in the presence of ambient sounds, or other non-target human speakers. In multi-speaker environments, where multiple speakers are talking simultaneously with a large amount of overlap, a significant performance degradation may occur with conventional ASR systems if trained by recordings of single talkers. This is called the cocktail party problem [1], and it occurs when the speeches of multiple speakers mix together, making it intractable for traditional single-speaker speech recognition models to understand what each person is independently saying. As an example, a dual speakers condition is shown in Figure 1. Two speakers are talking together, which is given by a single recording, and is denoted by $\mathbf{x}$. The multi-speaker ASR system is expected to generate the original text sets of what the two independent speakers are saying, denoted by ${\mathbf{y}}^{\left(1\right)}$ and ${\mathbf{y}}^{\left(2\right)}$. We use parenthesized and superscripted numbers for the speaker index on the variables, which is not to be confused with exponential notations. In Figure 1, the overlapped speech $\mathbf{x}$ in the middle is quite different from the first speaker’s voice (top left), as well as the second speaker’s (top right). Recovering the original texts, such as “Hello, this is me.” and “She goes to school.”, by listening to the mixed sound $\mathbf{x}$ only is very hard even for human listeners, and this is because adding two sounds may create unexpected artifacts and break the original sounding characteristics. However, these multi-speaker conditions are quite common, such as in ordinary conversations or in speaking with background human speakers. Therefore, multi-speaker speech recognition is required to build ASR systems that are utilizable in any circumstance. Conventional noise suppression techniques may not be applicable because they cannot separate sources with the same characteristics.

Many approaches have been proposed to address the challenge of multi-speaker ASR. Techniques such as deep clustering (DPCL) [2] and permutation invariant training (PIT) [3] have been proposed. These techniques are combined with a single-speaker ASR model by bypassing explicit separation outputs as inputs to the ASR model. These methods use either the hybrid ASR framework [4] or the end-to-end ASR framework [5,6]. The other types of approaches in multi-speaker ASR involve the utilization of speaker embeddings. Various models use speaker embedding as an additional input [7] or joint modeling with the speaker recognition model to create a multi-speaker ASR solution. Well-known methods include speaker-attributed training [8,9]. Speaker embedding techniques like x-vectors have been explored to improve the separation and the recognition of overlapped speech. However, these methods encounter challenges, including performance degradation due to speech signal distortion, high computational complexity, and the need for extensive training data. The persistent challenge of recognizing individual speaker speeches in overlapping speech highlights the need for more robust and improved multi-speaker ASR solutions.

To solve the aforementioned challenges, we propose a novel speaker-attributed training (SAT) method. The proposed method incorporates speaker embedding information as an additional input. The embedding information for each of the speakers in the training set is extracted as a numeric vector, and all of the embedding vectors are stacked to construct a speaker profile matrix. The profile matrix from the training dataset enables finding the similarities of the embedding vectors to those of the speakers of the input recordings in test conditions, and it helps to separately recognize the multiple speakers’ voices mixed in the input. Various speaker embedding vectors, such as i-vector [10], d-vector [11], and x-vector [12], are adopted. Furthermore, the proposed method efficiently reuses the decoder from the conventional SAT-ASR (speaker-attributed training for automatic speech recognition) [6] to eliminate the need for fully training the entire model.

Experimental results show that the proposed method improves existing multi-speaker speech recognition models based on the Transformer network [6] in terms of word error rate (WER) on the Librimix dataset [13]. The proposed SAT-ASR model demonstrates improved WERs in multi-speaker speech environments. Without a speaker profile matrix, we observed a 0.33% absolute improvement (3.9% relative improvement) in WER, and with a speaker profile matrix, a 0.95% absolute improvement (11.5% relative) was observed. Section 2 presents related work on speaker embeddings and multi-speaker speech recognition models. Section 3 introduces the proposed SAT multi-speaker ASR model and speaker profile. Section 4 evaluates the performance of this model. Finally, Section 5 concludes with a summary.

2. Related Work

This section describes the conventional methods to solve multi-speaker speech recognition problems. There are many abbreviations used in this manuscript, and all of them are listed at the end of Section 5 in the order of appearance.

2.1. Speaker Embedding

Speaker embedding techniques were used to extract the characteristics of speakers from speech signals, which have proven valuable in various tasks such as speaker verification [14], speaker diarization [15], speech synthesis [16], and speaker separation [17]. Similar to Word2Vec [18] and Wav2Vec [19,20], speaker embedding techniques generate fixed-dimensional numeric vectors from speech signals whose lengths vary over time. One of the more well-known speaker embedding vectors is the intermediate vector (i-vector) [10]. The i-vector is obtained by projecting a universal background model (UBM) of a speaker to characteristically coordinate vectors, which are trained to maximize the likelihood of the projection by exploiting unsupervised learning techniques. This projection maps high-dimensional representation from the UBM to a desired dimensional representation: the i-vector. These i-vectors are applied to speaker verification, facilitating decisions on whether speakers are the same or different [21] using a probabilistic linear discriminant analysis (PLDA) [22] classifier. The x-vector [12] utilizes deep neural networks (DNN) to extract the speaker characteristics from speeches. Another embedding method uses three-layer long short term memory (LSTM) with a projection exploiting a generalized end-to-end loss function [11]. The resultant embedding vectors are called d-vectors, and they have been shown to demonstrate a particularly better performance in text-dependent speaker recognition tasks on datasets with short utterances.

2.2. Conventional Multi-Speaker Speech Recognition Methods

Several methods have been proposed to generate multiple, speaker-specific texts from mixed recordings of multi-talkers with overlapped speeches. The conventional methods can be classified into the following four types.

(a)

A combination of single-speaker ASR models and existing acoustic source separation methods, such as deep clustering (DPCL) [2], permutation invariant training (PIT) [3,23], using x-vectors [24], self-supervised learning [25], and the joint learning of separation masks and speech recognition models [26].

(b)

Directly building a multi-speaker speech recognition system using mixed speech as the input and multiple text streams as learning targets [4,27,28,29].

(c)

The addition of speaker embedding vectors as indicators for the speakers in the input mixture [7,8,9,30] combined with the multi-speaker ASR system in (b) This is called speaker-attributed training for automatic speech recognition (SAT-ASR) [8].

(d)

Addition of a splitting encoder network that generates speaker-specific speech embedding vectors, using them as inputs to the single-speaker ASR decoder [5,6,31]. No retraining of a single-speaker ASR decoder is necessary.

As illustrated in Figure 2, Type (a) is implemented by combining existing source separation and ASR models, and no retraining or fine tuning is required. Its limitation is that the ASR performance degrades according to the distortion in the separated speech signals caused by the adopted monaural source separation method. The ASR performance is no better than the other types with fine tuning [4,5]. The second type constructs a single-input and multiple-output (SIMO) black box, with its inputs and outputs being mixed speech signals and multiple word sequences spoken by the individual speakers, respectively. This SIMO black box is usually implemented by either hybrid deep neural networks [4] or end-to-end ASR [5,6]. This method ensures decent multi-speaker ASR performance if a high enough amount of training data is given. However, each of the training samples should be mixed with all the other speaker samples to simulate all the possible combinations of talkers; thus, the training time may become $O\left(MN\right)$, where M is the number of speakers and N is the number of samples, while it is $O\left(N\right)$ for single-speaker ASR models. The third type, as shown in Figure 2-(c), is called speaker-attributed ASR [8] and appends speaker embedding vectors to the input speech features. The additional embedding vectors let the ASR be informed on the characteristics of overlapped speakers. It is expected to more separately train the multi-speaker ASR models, resulting in faster convergence, as well as improved performance [8,9]. It requires additional training of the speaker encoder with mixed-input recordings so the computational overhead of the whole training is the same order as Type (b)—$O\left(MN\right)$.

The last method, Type (d), was proposed to overcome the heavy-computation problem. The single-speaker ASR is constructed by a Transformer network [6], and its decoder is used without retraining. The “Split Encoder” shown in Figure 2-(d) was trained by a set of mixed-speech signals in a way that it can encode the mixed input into multiple representations, ${\mathbf{H}}^{\left(1\right)}$ and ${\mathbf{H}}^{\left(2\right)}$, to be compatible with the ASR decoder. The multiple text outputs, denoted by ${\mathbf{y}}^{\left(1\right)}$ and ${\mathbf{y}}^{\left(2\right)}$, where the parenthesized superscripts are speaker indices, were used as target labels, and only the split encoder was fully trained while the SISO ASR decoder was fixed. Training the ASR decoder takes $O\left(N\right)$ time, as does $O\left(MN\right)$ for the split encoder, but it takes much less time than the ASR decoder. Therefore, the computational overhead was greatly reduced when compared to the SIMO ASR approach.

This paper proposes a novel multi-speaker ASR method that takes advantages of Types (c) and (d): a combination of speaker-attributed training and encoder–decoder-based multi-speaker ASR. Speaker embedding vectors from the speaker encoder trained by the single-talker data in (c) were concatenated to the input of the split encoder of (d) to help splitting the mixed speech and inferring the encoder output to be closer to the single-speaker representations. The proposed method is computationally efficient, and permutation invariant training was not required because the speaker embedding vectors prevent permutation ambiguity. The implementation details of the proposed method are described in Section 3.

3. Elaborated Model of the Proposed Methods

3.1. Proposed Speaker-Attributed Training

The proposed model combines automatic speech recognition (ASR) using the encoder–decoder architecture of the Transformer network [6] and speaker-attributed training with speaker embedding vectors [7,8]. As shown in Figure 3, speaker encoder and decoder blocks of a Transformer network, ${\mathrm{Enc}}_{\mathrm{speaker}}$ and ${\mathrm{Dec}}_{\mathrm{speaker}}$, were trained using the ASR training dataset. The subscript ‘$\mathrm{speaker}$’ indicates that the encoder–decoder extracts speaker embedding vectors from the input recording. The training samples were single-talker utterances, and their targets were speaker identity labels that were used to obtain representations for the speaker differences in order to be maximally emphasized. The speaker embedding block consists of the trained ${\mathrm{Enc}}_{\mathrm{speaker}}$ and ${\mathrm{Dec}}_{\mathrm{speaker}}$, which is represented as

$$\begin{array}{ccc}\hfill \mathbf{X}& =& \{{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_T\},\hfill \end{array}$$

(1)

$$\begin{array}{ccc}\hfill \mathbf{E}=\{{\mathbf{e}}_1,{\mathbf{e}}_2,\dots ,{\mathbf{e}}_T\}& =& {\mathrm{Enc}}_{\mathrm{speaker}}\left(\mathbf{X}\right),\hfill \end{array}$$

(2)

$$\begin{array}{ccc}\hfill {\mathbf{Q}}_{\mathrm{mix}}=\{{\mathbf{q}}_1,{\mathbf{q}}_2,\dots ,{\mathbf{q}}_T\}& =& {\mathrm{Dec}}_{\mathrm{speaker}}\left(\mathbf{E}\right),\hfill \end{array}$$

(3)

where T is the number of the feature vectors from the time-varying input recording, ${\mathbf{x}}_t$ is the extracted feature at the analysis frame index t, ${\mathbf{e}}_t$ is the encoder output, and ${\mathbf{q}}_t$ is the extracted speaker embedding vector at the index t. The embedding vector ${\mathbf{q}}_t$ is also referred to as speaker query vector [8,32] because it identifies the speaker in the input speech. The capital notations, such as $\mathbf{X}$, $\mathbf{E}$, and ${\mathbf{Q}}_{\mathrm{mix}}$, represent the set of all T vectors, and they are often implemented by matrices. We added ‘$\mathrm{mix}$’ to $\mathbf{E}$ as a subscript to imply that this set of speaker embedding vectors is used by the ASR encoder ${\mathrm{Enc}}_{\mathbf{mix}}$.

In the case of the d-vector method [33,34], the extraction was autoregressive, so the embedding vector at a time was dependent on the previous vectors only, and this can be expressed by

$${\mathbf{q}}_t={\mathrm{Dec}}_{\mathrm{speaker}}^{(d-vec)}\left(\left\{{\mathbf{q}}_1,\dots {\mathbf{q}}_{t-1}\right\},{\mathbf{e}}_t\right),t=1,\dots ,T.$$

(4)

As show in Figure 3, the ASR encoder was divided into three encoders with different roles. The acoustic feature $\mathbf{X}$ and the speaker embedding ${\mathbf{Q}}_{\mathrm{mix}}$ were concatenated and were called vertical stacking [7]. The first encoder ${\mathrm{Enc}}_{\mathrm{mix}}$ maps the concatenated input to a higher-dimensional representation ${\mathbf{H}}_{\mathrm{mix}}$,

$${\mathbf{H}}_{\mathrm{mix}}={\mathrm{Enc}}_{\mathrm{mix}}\left(\mathbf{X},{\mathbf{Q}}_{\mathrm{mix}}\right).$$

(5)

The role of ${\mathrm{Enc}}_{\mathrm{mix}}$ is in emphasizing the information of the speakers by merging the speaker embedding ${\mathbf{Q}}_{\mathrm{mix}}$ and $\mathbf{X}$. From the higher-dimensional representation ${\mathbf{H}}_{\mathrm{mix}}$, the speaker-differentiating encoder ${\mathrm{Enc}}_{\mathrm{diff}}^{\left(s\right)}$ extracted the speech of each speaker,

$${\mathbf{H}}^{\left(s\right)}={\mathrm{Enc}}_{\mathrm{diff}}^{\left(s\right)}\left({\mathbf{H}}_{\mathrm{mix}}\right),s=1,\dots ,S,$$

(6)

where s and S represent the speaker index and the total number of speakers, respectively. As shown in Figure 3, $S=2$. ${\mathrm{Enc}}_{\mathrm{diff}}^{\left(s\right)}$ splits the mixed features of S speakers so that single-speaker information remains in each ${\mathbf{H}}^{\left(s\right)}$. The last ${\mathrm{Enc}}_{\mathrm{rec}}$ transforms each ${\mathbf{H}}^{\left(s\right)}$ into the embeddings ${\mathbf{G}}^{\left(s\right)}$ to be compatible with the pretrained decoder ${\mathrm{Dec}}_{\mathrm{rec}}$. The encoder weights were shared over all the input ${\mathbf{H}}^{\left(s\right)}$. Finally, ${\mathrm{Dec}}_{\mathrm{rec}}$ maps each ${\mathbf{G}}^{\left(s\right)}$ to the output ${\mathbf{y}}^{\left(s\right)}$, which is typically a word sequence for ASR. The pretrained decoder ${\mathrm{Dec}}_{\mathrm{rec}}$ was fixed and shared over all of the input.

$$\begin{array}{ccc}\hfill {\mathbf{G}}^{\left(s\right)}& =& {\mathrm{Enc}}_{\mathrm{rec}}\left({\mathbf{H}}^{\left(s\right)}\right),\hfill \end{array}$$

(7)

$$\begin{array}{ccc}\hfill {\mathbf{y}}^{\left(s\right)}& =& {\mathrm{Dec}}_{\mathrm{rec}}\left({\mathbf{G}}^{\left(s\right)}\right).\hfill \end{array}$$

(8)

The three-stage encoder design has several advantages. The recognition encoder, ${\mathrm{Enc}}_{\mathrm{rec}}$, is focused on matching the previous encoder output to the recognition decoder ${\mathrm{Dec}}_{\mathrm{rec}}$. Their weights are shared so that the number of trainable parameters can be reduced. The speaker splitting is performed by ${\mathrm{Enc}}_{\mathrm{diff}}$, and merging the speaker embedding vectors and speech features is performed by ${\mathrm{Enc}}_{\mathrm{mix}}$ only. By assigning different roles, it can provide faster training and improved performance. Another advantage is that the pretrained ASR decoder and speaker encoder–decoder is used without retraining. Moreover, those modules are trained by single-talker recordings, and only encoders require multi-talker recordings for fine tuning, as shown by the dotted box in Figure 3.

The whole procedure of the proposed speaker-attributed training is shown in Algorithms 1–3. The Procedure Forward in Algorithm 1 is a step-by-step illustration of the forward inference in Figure 3. The inputs are a set of mixed feature vectors $\mathbf{X}$ in Equation (1), where the parameter sets are ${\Theta }_0$ and ${\Theta }_1$ and the outputs are the predicted word sequences $\widehat{\mathbf{Y}}$ for the given input $\mathbf{X}$. ${\Theta }_0$ consists of a speaker encoder and decoder to extract speaker embedding vectors, i.e., ${\mathrm{Enc}}_{\mathrm{speaker}}$ and ${\mathrm{Dec}}_{\mathrm{speaker}}$, and the pretrained speech recognition decoder is ${\mathrm{Dec}}_{\mathrm{rec}}$. ${\Theta }_1$ is a set of ${\mathrm{Enc}}_{\mathrm{mix}}$, ${\mathrm{Enc}}_{\mathrm{diff}}^{(1\dots S)}$, and ${\mathrm{Enc}}_{\mathrm{rec}}$, which are required by Equations (5)–(7). The Procedure Backward in Algorithm 2 requires the predicted and target word sequences $\widehat{\mathbf{Y}}$ and $\mathbf{Y}$. It first computes the loss between $\widehat{\mathbf{Y}}$ and $\mathbf{Y}$, and then it performs a single backpropagation learning step to minimize the loss (denoted by L).

The Procedure SAT in Algorithm 3 defines the whole training process. It requires an input of mixed features $\mathbf{X}$, target word sequences $\mathbf{Y}$, and parameter set ${\Theta }_0$. These inputs are fixed throughout the entire training process. The output of Procedure SAT, ${\Theta }_1$, is a set of a mixed encoder, speaker-differentiation encoder, and recognition encoder. In Line 2 of Algorithm 3, they are created by an appropriate initialization method. In Lines 4–5, the input features $\mathbf{X}$ is passed through the network constructed by the parameters ${\Theta }_0$ and ${\Theta }_1$ by calling a sub-procedure Forward, and set ${\Theta }_1$ is updated by calling Backward. These are repeated until the convergence of the training loss L. The detailed implementation is publicly available at https://github.com/craft8244/MSSR, accessed on 17 August 2024. Although Figure 3 shows architectures for two-speaker mixing cases, the proposed method can be applied to more than two speakers cases by replicating ${\mathrm{Enc}}_{\mathrm{diff}}$ only. In the case of the other encoders, ${\mathrm{Enc}}_{\mathrm{mix}}$ and ${\mathrm{Enc}}_{\mathrm{rec}}$, only a single instance is required, so there is no architectural change according to the number of speakers. Once adequate training samples with the desired number of speakers are given, the proposed method can be easily applied.

Algorithm 1 Speaker-attributed forward inference

Require: $\mathbf{X}=\left[{\mathbf{x}}_1\dots {\mathbf{x}}_T\right]$, ${\Theta }_0$, ${\Theta }_1$▹ overlapped speech and transformer encoder/decoders 1: procedure Forward($\mathbf{X}$, ${\Theta }_0$, ${\Theta }_1$): 2:     $\left({\mathrm{Enc}}_{\mathrm{speaker}},{\mathrm{Dec}}_{\mathrm{speaker}},{\mathrm{Dec}}_{\mathrm{rec}}\right)\leftarrow {\Theta }_0$ 3:     $\left({\mathrm{Enc}}_{\mathrm{mix}},{\mathrm{Enc}}_{\mathrm{diff}}^{(1\dots S)},{\mathrm{Enc}}_{\mathrm{rec}}\right)\leftarrow {\Theta }_1$           ▹ fetch transformer parameters 4:     $\mathbf{E}\leftarrow {\mathrm{Enc}}_{\mathrm{speaker}}\left(\mathbf{X}\right)$                  ▹ generate speaker encoder output 5:     ${\mathbf{Q}}_{\mathrm{mix}}\leftarrow {\mathrm{Dec}}_{\mathrm{speaker}}\left(\mathbf{E}\right)$     ▹ convert to a set of multi-speaker embedding vectors 6:     $(\mathbf{X},{\mathbf{Q}}_{\mathrm{mix}})$            ▹ concatenate embedding and overlapped speech 7:     ${\mathbf{H}}_{\mathrm{mix}}\leftarrow {\mathrm{Enc}}_{\mathrm{mix}}\left(\mathbf{X},{\mathbf{Q}}_{\mathrm{mix}}\right)$              ▹ map to high-dimensional features 8:     for $s=1,\dots ,S$ do: 9:         ${\mathbf{H}}^{\left(s\right)}\leftarrow {\mathrm{Enc}}_{\mathrm{diff}}^{\left(s\right)}\left({\mathbf{H}}_{\mathrm{mix}}\right)$                 ▹ extract single-speaker features 10:         ${\mathbf{G}}^{\left(s\right)}\leftarrow {\mathrm{Enc}}_{\mathrm{rec}}\left({\mathbf{H}}^{\left(s\right)}\right)$             ▹ convert to decoder-compatible features 11:         ${\widehat{\mathbf{y}}}^{\left(s\right)}\leftarrow {\mathrm{Dec}}_{\mathrm{rec}}\left({\mathbf{G}}^{\left(s\right)}\right)$            ▹ generate speech recognition outputs 12:     end for 13:     return $\widehat{\mathbf{Y}}=\left\{{\widehat{\mathbf{y}}}^{\left(1\right)},\dots {\widehat{\mathbf{y}}}^{\left(S\right)}\right\}$ 14: end procedure

Algorithm 2 Backward learning

Require: $\widehat{\mathbf{Y}}$, $\mathbf{Y}$, ${\Theta }_1$   ▹ predicted and target word sequences, and parameters to update 1: procedure Backward($\widehat{\mathbf{Y}}$, $\mathbf{Y}$, ${\Theta }_1$): 2:     $L\leftarrow$ LOSS$(\widehat{\mathbf{Y}},\mathbf{Y})$  ▹ compute the loss between estimated and true word sequences 3:     ${\Theta }_1\leftarrow$ Backprop$({\Theta }_1,L)$    ▹ perform backpropagation learning to update parameters minimizing L 4:     return L, ${\Theta }_1$ 5: end procedure

Algorithm 3 Speaker-attributed training (SAT)

Require: $\mathbf{X}=\left[{\mathbf{x}}_1\dots {\mathbf{x}}_T\right]$, $\mathbf{Y}=\left[{\mathbf{y}}^{\left(1\right)}\dots {\mathbf{y}}^{\left(S\right)}\right]$, ${\Theta }_0=\left({\mathrm{Enc}}_{\mathrm{speaker}},{\mathrm{Dec}}_{\mathrm{speaker}},{\mathrm{Dec}}_{\mathrm{rec}}\right)$ ▹ overlapped speech, multiple target word sequences, pretrained transformer encoder/decoders 1: procedure SAT($\mathbf{X}$, $\mathbf{Y}$, ${\Theta }_0$): 2:      ${\Theta }_1\leftarrow$INIT$\left({\mathrm{Enc}}_{\mathrm{mix}},{\mathrm{Enc}}_{\mathrm{diff}}^{(1\dots S)},{\mathrm{Enc}}_{\mathrm{rec}}\right)$   ▹ Initialize multi-speaker encoders 3:     repeat 4:          $\widehat{\mathbf{Y}}\leftarrow$Forward$(\mathbf{X},{\Theta }_0,{\Theta }_1)$ 5:          L, ${\Theta }_1\leftarrow$Backward$(\widehat{\mathbf{Y}},\mathbf{Y},{\Theta }_1)$ 6:     until L converges 7:     return ${\Theta }_1$ 8: end procedure

3.2. Speaker-Attributed Training Using Speaker Profiles

Because speaker embedding vectors are provided as the input to ${\mathrm{Enc}}_{\mathrm{mix}}$, they should be as relevant to the speakers as possible in order to obtain optimal splitting performance. However, because ${\mathrm{Enc}}_{\mathrm{speaker}}$ and ${\mathrm{Dec}}_{\mathrm{speaker}}$ were trained by single-talker recordings, the extracted speaker embedding vectors may not have reliably modeled the mixed speakers. Therefore, we adopted attention-weighted averaging of the speaker profiles [8,32,35] to better match the input to the speaker embedding vectors in the target profile.

Through using a speaker encoder–decoder on the training data, speaker embedding vectors were extracted, and those embedding vectors were clustered into a predefined number of profile vectors [32], which are denoted as $\mathbf{P}=\{{\mathbf{p}}_1,{\mathbf{p}}_2,\dots ,{\mathbf{p}}_K\}$, where K is the number of profile vectors. In this paper, $\mathbf{P}$ was implemented by a matrix, where each column represents a profile embedding vector, and the number of profiles used was 40. For the input query vector ${\mathbf{q}}_t$, its closeness to profile k was computed by a cosine similarity between two vectors as follows [32]:

$$b_{t,k}={\displaystyle \frac{{\mathbf{q}}_t·{\mathbf{p}}_k}{\parallel {\mathbf{q}}_t\parallel \parallel {\mathbf{p}}_k\parallel }},t=1,2,\dots T,$$

(9)

where the operator · is an inner product, t is the frame index over time (in our work, it was extracted at every 10 milliseconds), k is the profile vector index in the profile matrix $\mathbf{P}$, and T is the number of speaker embedding vectors. In our implementation, the inner product ${\mathbf{q}}_t·{\mathbf{p}}_k$ was efficiently computed by performing ${\mathbf{Q}}^{\mathrm{T}}\mathbf{P}$, where $\mathrm{T}$ is the matrix transpose operator that generates $T\times K$ inner products in a single step with a GPU. The cosine similarity between embedding vector ${\mathbf{q}}_t$ and profile vector ${\mathbf{p}}_k$, $b_{t,k}$ ranged from −1 to 1, so the simple linear weighting may be greatly varied. Therefore, $b_{t,k}$ was scaled by a softmax normalization so that the resultant attention weight ${\beta }_{t,k}$ ranged from 0 and 1:

$${\beta }_{t,k}={\displaystyle \frac{exp\left(b_{t,k}\right)}{{\sum }_{j=1}^{K}exp\left(b_{t,j}\right)}}.$$

(10)

The attention-weighted speaker profile ${\tilde{\mathbf{q}}}_t$ was calculated as a linear combination of the profile vectors weighted by ${\beta }_{t,k}$ as

$$\begin{array}{ccc}\hfill {\tilde{\mathbf{q}}}_t& =& \sum _{k=1}^K{\beta }_{t,k}{\mathbf{p}}_k,\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hfill {\tilde{\mathbf{Q}}}_{mix}& =& \left\{{\tilde{\mathbf{q}}}_1,{\tilde{\mathbf{q}}}_2,\dots ,{\tilde{\mathbf{q}}}_T\right\}.\hfill \end{array}$$

(12)

The speaker embedding set ${\mathbf{Q}}_{mix}$ in Equation (5) was replaced with ${\tilde{\mathbf{Q}}}_{mix}$ in Equation (12). Once the profile matrix $\mathbf{P}$ was prepared, only cosine similarity computation was required, resulting in minimal computational overhead during the inference step. Figure 4 illustrates the speaker-attributed ASR using speaker profiles. The speaker profiling block was added to replace the speaker embedding vector with the weighted average of the profile vectors. The speaker profiling was expected to convert the unknown speaker embedding into a linear combination of known speaker embedding vectors; therefore, ${\mathrm{Enc}}_{\mathrm{mix}}$ may have better split the speaker information into ${\mathbf{H}}^{\left(1\right)}$ and ${\mathbf{H}}^{\left(2\right)}$.

Procedure Beta in Algorithm 4 is a step-by-step description of the attention-weighted speaker embedding vector derivation in Equations (9)–(11). The computed similarities represent the attention of the $\mathbf{q}$ to the enrolled profile vectors. It computes cosine similarities for all embedding vectors in the profile matrix $\mathbf{P}$, and it replaces the speaker embedding vector $\mathbf{q}$ with the linear combination of the profile embedding vectors. Procedure Forward in Algorithm 5 is a forward inference with a profile matrix. Most of the steps are identical to Algorithm 1 except that the speaker embedding ${\mathbf{Q}}_{\mathrm{mix}}$ of the mixed-speech features was replaced with the attention-weighted embedding vectors, as shown in Lines 6–9. There was no change in the backpropagation steps and training iterations, so Algorithms 2 and 3 can be used without any modification. The detailed implementation is also publicly available at https://github.com/craft8244/MSSR, accessed on 17 August 2024.

Algorithm 4 Attention-weighted speaker embedding vector extraction

Require: $\mathbf{q}$, ${\Theta }_1$, $\mathbf{P}$    ▹ speaker embedding vector, speaker profile matrix 1: procedure Beta($\mathbf{q}$, $\mathbf{P}$): 2:     $K\leftarrow \left|\mathbf{P}\right|$       ▹ number of profile vectors 3:     for $k=1,\dots ,K$ do: 4:         $b_k\leftarrow \mathbf{q}·{\mathbf{p}}_k/\left(\parallel \mathbf{q}\parallel \parallel {\mathbf{p}}_k\parallel \right)$      ▹ compute cosine similarity 5:     end for 6:     for $k=1,\dots ,K$ do: 7:         ${\beta }_k\leftarrow exp\left(b_{t,k}\right)/{\sum }_{j=1}^{K}exp\left(b_{t,j}\right)$      ▹ softmax normalization 8:     end for 9:     return $\tilde{\mathbf{q}}=\left({\sum }_{k=1}^K{\beta }_k{\mathbf{p}}_k\right)$    ▹ return attention-weighted speaker embedding 10: end procedure

Algorithm 5 Forward inference using speaker profile

Require: $\mathbf{X}=\left[{\mathbf{x}}_1\dots {\mathbf{x}}_T\right]$, ${\Theta }_0$, ${\Theta }_1$, $\mathbf{P}$    ▹ overlapped speech and transformer encoder/decoders, speaker profile matrix 1: procedure Forward($\mathbf{X}$, ${\Theta }_0$, ${\Theta }_1$, $\mathbf{P}$): 2:     $\left({\mathrm{Enc}}_{\mathrm{speaker}},{\mathrm{Dec}}_{\mathrm{speaker}},{\mathrm{Dec}}_{\mathrm{rec}}\right)\leftarrow {\Theta }_0$ 3:     $\left({\mathrm{Enc}}_{\mathrm{mix}},{\mathrm{Enc}}_{\mathrm{diff}}^{(1\dots S)},{\mathrm{Enc}}_{\mathrm{rec}}\right)\leftarrow {\Theta }_1$      ▹ fetch transformer parameters 4:     $\mathbf{E}\leftarrow {\mathrm{Enc}}_{\mathrm{speaker}}\left(\mathbf{X}\right)$    ▹ generate a matrix of multi-speaker encoder outputs 5:     $\left[{\mathbf{q}}_1\dots {\mathbf{q}}_T\right]\leftarrow {\mathrm{Dec}}_{\mathrm{speaker}}\left(\mathbf{E}\right)$   ▹ convert to a matrix of embedding vectors 6:     for $t=1,\dots ,T$ do: 7:         ${\tilde{\mathbf{q}}}_t\leftarrow Beta({\mathbf{q}}_t,\mathbf{P})$        ▹ attention-weighting 8:     end for 9:      ${\tilde{\mathbf{Q}}}_{mix}=\left[{\tilde{\mathbf{q}}}_1\dots {\tilde{\mathbf{q}}}_T\right]$    ▹ construct attention-weighted embedding matrix 10:     ${\mathbf{H}}_{\mathrm{mix}}\leftarrow {\mathrm{Enc}}_{\mathrm{mix}}\left(\mathbf{X},{\tilde{\mathbf{Q}}}_{\mathrm{mix}}\right)$     ▹ map to high-dimensional features 11:     for $s=1,\dots ,S$ do: 12:         ${\mathbf{H}}^{\left(s\right)}\leftarrow {\mathrm{Enc}}_{\mathrm{diff}}^{\left(s\right)}\left({\mathbf{H}}_{\mathrm{mix}}\right)$      ▹ extract single-speaker features 13:         ${\mathbf{G}}^{\left(s\right)}\leftarrow {\mathrm{Enc}}_{\mathrm{rec}}\left({\mathbf{H}}^{\left(s\right)}\right)$    ▹ convert to decoder-compatible features 14:         ${\widehat{\mathbf{y}}}^{\left(s\right)}\leftarrow {\mathrm{Dec}}_{\mathrm{rec}}\left({\mathbf{G}}^{\left(s\right)}\right)$     ▹ generate speech recognition outputs 15:     end for 16:     return $\widehat{\mathbf{Y}}=\left\{{\widehat{\mathbf{y}}}^{\left(1\right)},\dots {\widehat{\mathbf{y}}}^{\left(S\right)}\right\}$ 17: end procedure

4. Experimental Performance Evaluation

The proposed method was evaluated by its speech recognition performance on the LibriMix dataset [13], where each audio sample contains only two speakers. Specifically, the speech recognition performance was evaluated based on the speaker embeddings from the speech profiles used in the proposed method, as well as the number of utterances extracted for the profile. Furthermore, we compared the speech recognition performance across three different systems using speaker embeddings and speaker profiles: single-speaker ASR, existing multi-speaker ASR, and the proposed SAT-ASR methods.

4.1. Experimental Setup

The LibriMix dataset is an open-source, single-channel, and multi-speaker speech dataset derived from LibriSpeech [36]. It is composed of mixtures involving two or three speakers, along with ambient noise samples from WHAM! [37]. This dataset is constructed by mixing single-speaker recordings from LibriSpeech with partial overlap and integrated noise. The dataset is divided into two primary collections [13]: Libri2Mix, which features two speakers; and Libri3Mix, with three speakers. We carried out experiments on Libri2Mix only because two-speaker mixing is much more common in real environments. Each of these subsets include versions with and without WHAM! noise, leading to ‘clean’ data free from noise and ‘both’ data containing noise. There are four versions of the dataset available, differentiated by two sampling rates (16 kHz and 8 kHz) and two modes (min and max). In min mode, the mixture concludes at the end of the shortest utterance, while in max mode, the shortest utterance is extended to match the length of the longest one. For our proposed model, which is designed for scenarios with two speakers, we utilize the Libri2Mix subset of LibriMix. We specifically chose the ‘clean’ dataset without noise at a 16 kHz sampling rate and in max mode to maximize the inclusion of speech data for training.

Table 1. Statistics of the evaluation datasets. The number of utterances, number of speakers, and total duration of recordings in hours are provided for each set. The Train set was used for training the models, the Dev set was for hyperparameter tuning, and the Test set was for the final evaluation. There were no overlaps in the samples or the speakers within the dataset. All the samples were mixed recordings of two speakers from Libri2Mix.

Dataset	No. Utterances	No. Speakers	Hours
Train	64,700	1172	270
Dev	3000	40	11
Test	3000	40	11

presents the statistics of the evaluation datasets. The training dataset consists of 64,700 utterances, amounting to approximately 270 h. The development dataset consists of 3000 utterances, totaling 11 h. Similarly, the test set includes 3000 utterances, also spanning 11 h, and is completely independent from the training and development sets. The dataset includes the following six different rates of speech overlap: 0%, 20%, 40%, 60%, 80%, and 100%. The maximum duration of each utterance is 15 s. The training data, development data, and test data consist of 1172, 40, and 40 independent speakers, respectively.

In Section 4.3, the evaluation results from the three types of speaker embeddings, i-vector, d-vector, and x-vector, are shown. The intermediate vector (i-vector) extractor [12] was constructed using Gaussian mixture models (GMM) based on the universal background model (UBM). We extracted 400-dimensional speaker embeddings using the Kaldi toolkit [38,39]. The pretrained i-vector extractor was obtained from the Kaldi official website [38]. The x-vector extractor [12], based on a deep neural network, was utilized to extract 512-dimensional speaker embedding vectors from the input utterances. We employed a pretrained x-vector extractor from the Kaldi official website. Both of these models were trained on the augmented VoxCeleb 1 and 2 datasets [40]. The input features for both extractors were 30-dimensional mel-frequency cepstral coefficients (MFCCs), with a frame length of 25 milliseconds and a shift size of 10 milliseconds. The x-vectors were only extracted from voice frames, which were selected by an energy-based voice activity detector. $L_2$-normalized speaker embeddings were then used as an additional input to the speech recognition decoder. The d-vector extractor [11] was based on a recurrent neural network implemented in PyTorch [41]. We extracted 256-dimensional speaker embedding vectors. The input features for the extractor were 40-dimensional mel-scale log spectrograms, with a frame length of 25 milliseconds and a frame shift size of 10 milliseconds. The extractor was configured with a three-layer LSTM, where each layer had a hidden dimension of 256.

4.2. ASR Model Training

All proposed multi-speaker speech recognition models were implemented by the ESPnet framework [42] with the PyTorch [41] backend. The acoustic features consisted of 80-dimensional mel-scale log filterbank energies, including pitch features, deltas, and delta-deltas. The input features were created by concatenating the speaker embedding vector with the acoustic features. Consequently, the dimension of the input feature depended on the type of speaker embedding vectors adopted. For instance, in the case of i-vectors, the input dimension became 480 (400 for the speaker embedding and 80 for the acoustic feature). In the case of x-vectors, the dimension was 592 (512 + 80), and it was 336 (256 + 80) for the d-vectors. In the Transformer-based multi-speaker decoder ASR model, there are 18 layers in the conventional method [43,44].

For the proposed methods, ${\mathrm{Enc}}_{\mathrm{mix}}$ acts as the CNN (convolutional neural network) embedding layer. ${\mathrm{Enc}}_{\mathrm{diff}}$ and ${\mathrm{Enc}}_{\mathrm{rec}}$ consist of four and eight Transformer layers, respectively. For all Transformers, the attention size was set to 256, the number of attention heads to 4, and the fully-connected feed-forward network size to 2048. The training of the Transformer model was conducted with the Adam optimizer and Noam learning rate decay, as outlined in [3]. The backend ASR, ${\mathrm{Dec}}_{\mathrm{rec}}$, was initialized with a pretrained single-speaker speech recognition model from the ESPnet recipe of the LibriSpeech dataset. During training, it was kept frozen for the first 15 epochs to ensure stability. At the end of all ASR model training, a Transformer language model from the ESPnet recipe of the LibriSpeech dataset was employed on the pretrained single-speaker LibriSpeech dataset model without fine tuning. Similarly, as introduced in Section 4.1, all speaker embedding extractors were also used without additional training on the pretrained models from the VoxCeleb 1 and 2 speaker recognition datasets [40].

To train the baseline ASR model and the proposed models, we used four units of NVIDIA RTX TITAN with a 24 GB GPU memory. The single-speaker ASR model based on Transformer architecture has 113 million parameters, and its training data consist of 460 h of train-clean-100 and train-clean-360 subsets of LibriSpeech [36]. In terms of training time, it required 223 h, which is equivalent to 892 GPU hours because four GPUs were used. The baseline multi-speaker ASR [43,44] has 90.6 million parameters, and 30.6 million of them were taken from a single-speaker ASR Transformer model. The remaining 60 million parameters were trained by multi-speaker utterances. The multi-speaker training set consisted of 270 h, as shown in Table 1, and it took 320 GPU hours to train the multi-speaker modules. For the proposed SAT methods, 0, 11.2, and 4.2 million additional parameters were required for the i-, d-, and x-vectors, respectively. They took 320, 336, and 328 additional GPU hours to train. The number of trainable parameters and the total training hours are compared in detail in Table 2.

4.3. Evaluation Results and Discussions

To determine the best-performing speaker embedding and profile settings for the proposed SAT-ASR model, we evaluated the performance of various profile and speaker embedding configurations. The first set of experiments confirmed performance changes depending on the number of utterances used per profile. Figure 5 shows the word error rate (WER) variations according to the number of profile utterances. In this experiment, the SAT-ASR model using x-vector speaker profiles was used for evaluation. The x-axis is the number of utterances per profile. When the number of utterances (single-speaker samples) for each profile was 1, the lowest performance was 8.61% WER. When the number of utterances per profile was set to 5, 10, and 20, the performance improved to 7.83%, 7.57%, and 7.48%, respectively. Performance increased as the number of utterances for each profile increased, but it remained the same after 20. Based on these results, 20 utterances per profile were used in the speech recognition performance evaluation.

Table 2 compares the proposed SAT-ASR models with conventional models. The number of trainable parameters, training time, and test WER are listed. For the single-speaker automatic speech recognition (SS-ASR) model, we adopted a simple Transformer-based architecture [44] because the proposed SAT-ASR methods require pretrained decoders from the SS-ASR. Its implementation recipe was retrieved from ESPnet repository at https://github.com/espnet/espnet/tree/master/egs2/librispeech/asr1, accessed on 20 July 2024, where WER was reported for the single-speaker input and the test-clean set of LibriSpeech was 2.5. The SS-ASR model has 113 million (denoted by `M’ in Table 2) trainable parameters, and the total training data consists of 460 h of the train-clean subset of LibriSpeech [36]. It took 892 GPU hours to train all the parameters from scratch. As shown in Table 1, the test data were 3000 utterances of the test-clean subset of Libri2Mix generated by LibriMix. The evaluated WER was 76.9%, which is significantly worse than the LibriSpeech test dataset. The huge degradation was due to the mismatch between the stingle-speaker ASR model and multi-speaker input recordings. The second model uses a DPRNN-TasNet (dual-path recurrent neural network-based time-domain audio separation network) [45] to separate the speakers in the mixed-speech input. After separation, the word sequence of each output was generated using the SS-ASR, and the WER of each channel was measured. Because the pretrained SS-ASR was used, only a DPRNN-TasNet was required to be trained, so the number of trainable parameters was only 2.6 million. It took 212 GPU hours to train the DPRNN-TasNet model measured by the same NVDIAI GTX Titan GPU. The WER on the test-clean set of Libri2Mix was 13.6%, which a significant improvement on SS-ASR.

Table 2. Performance comparison of the baseline model and the proposed models in terms of the WER and their relative improvements (shown in parentheses) from the baseline. For the compact notation, the following abbreviations were used: SS-ASR (single-speaker automatic speech recognition); MS-ASR (multi-speaker automatic speech recognition); SAT (speaker-attributed training); and DPRNN-TasNet (dual-path recurrent neural network-based time-domain audio separation network). The second and third columns show the number of parameters to be trained (where `M’ stands for million) and the time required by a single NVIDIA GTX Titan GPU to train those parameters, respectively. The forth column shows the word error rates on the test set.

Model		Trainable	Training Time	Test
		Parameters	(GPU Hours)	WER (%)
SS-ASR [44]		113 M	892	76.90
DPRNN-TasNet + SS-ASR [45]		2.6 M	212	13.60
MS-ASR [6]		60 M	320	8.43 (-)
MS-ASR + SAT	i-vector	60 M	320	8.50 (−0.8)
(proposed)	d-vector	71.2 M	336	8.32 (1.3)
	x-vector	64.2 M	328	8.10 (3.9)
MS-ASR + SAT	i-vector	60 M	320	8.27 (1.9)
+ speaker profile	d-vector	71.2 M	336	8.18 (3.0)
(proposed)	x-vector	64.2 M	328	7.48 (11.2)

Table 2. Performance comparison of the baseline model and the proposed models in terms of the WER and their relative improvements (shown in parentheses) from the baseline. For the compact notation, the following abbreviations were used: SS-ASR (single-speaker automatic speech recognition); MS-ASR (multi-speaker automatic speech recognition); SAT (speaker-attributed training); and DPRNN-TasNet (dual-path recurrent neural network-based time-domain audio separation network). The second and third columns show the number of parameters to be trained (where `M’ stands for million) and the time required by a single NVIDIA GTX Titan GPU to train those parameters, respectively. The forth column shows the word error rates on the test set.

Model		Trainable	Training Time	Test
		Parameters	(GPU Hours)	WER (%)
SS-ASR [44]		113 M	892	76.90
DPRNN-TasNet + SS-ASR [45]		2.6 M	212	13.60
MS-ASR [6]		60 M	320	8.43 (-)
MS-ASR + SAT	i-vector	60 M	320	8.50 (−0.8)
(proposed)	d-vector	71.2 M	336	8.32 (1.3)
	x-vector	64.2 M	328	8.10 (3.9)
MS-ASR + SAT	i-vector	60 M	320	8.27 (1.9)
+ speaker profile	d-vector	71.2 M	336	8.18 (3.0)
(proposed)	x-vector	64.2 M	328	7.48 (11.2)

The third row is the Transformer-based multi-speaker automatic speech recognition (MS-ASR) method [6]. This model corresponds to Figure 2-(b), which uses a split encoder for the multi-speaker speech inputs. It requires training a split encoder (${\mathrm{Enc}}_{\mathrm{mix}}$ in Figure 3), speaker differentiation encoder (${\mathrm{Enc}}_{\mathrm{diff}}$), and recognition encoder (${\mathrm{Enc}}_{\mathrm{rec}}$). The total number of trainable parameters was about 60 million, and they were trained with the train-clean subset of Libri2Mix, whose size is 64,700 utterances and 270 h of mixed speech, as shown in Table 1. The training time was 320 GPU hours, excluding the time for the pretrained ${\mathrm{Enc}}_{\mathrm{speaker}}$, ${\mathrm{Dec}}_{\mathrm{speaker}}$, and ${\mathrm{Dec}}_{\mathrm{rec}}$ in Figure 3. The evaluated WER on the same test set was 8.43%, which represents a significant improvement over the other conventional methods.

The next three rows are the results of the proposed method described in Section 3.1 and correspond to Figure 3. Three different types of speaker embedding vectors were concatenated with the input features of mixed speech to emphasize the information of the speakers. Then, the proposed speaker-attributed training (SAT) technique in Algorithm 3 was applied to the baseline MS-ASR. The number of trainable parameters for the i-vectors was the same as that of the baseline MS-ASR because there were no additional layers. In the case of the d- and x-vectors, 11.2 and 4.2 million trainable parameters were added. The training times with the same train-clean set of Libri2Mix were 320, 336, and 328 GPU hours, respectively. The increased training time was less than 5% compared to the baseline. However, these methods also require the decoders of the pretrained SS-ASR model so one should also consider the overhead of training SS-ASR if it is not available. The WER of the i-vectors on the test set was 8.5%, which is worse than the baseline’s WER of 8.43%. The d-vectors’ WER was 8.32%, whose relative improvement was 1.3% from the baseline. For the x-vectors, a WER of 8.1% and 3.9% relative improvement were obtained, and they achieved the best results among the methods with speaker-attributed training using speaker embedding vectors directly.

The bottom three rows present the results of the method described in Section 3.2 and are shown in Figure 4. The same speaker embedding vectors were used, but the extracted vectors were replaced by the linear combinations of the profile embedding vectors from the 20 selected utterances. The differences from the method in Section 3.1 were due to the attention weighting provided by Algorithm 4 and the forward inference by Algorithm 5. The storage size and computation time of the profile matrix was almost negligible so the number of trainable parameters and training times were almost the same as SAT without profile weighting. The evaluated WERs on the test set were 8.27%, 8.18%, and 7.48% for the i-, d-, and x-vectors, respectively. All of the WERs were improved with the addition of speaker profiles. In particular, the x-vectors achieved a 11.2% relative improvement from the baseline, which was the best among all the cases.

Table 3. An example of the output word sequences generated by the proposed method. The input is a mixed speech of two speakers saying Label-A and Label-B with temporal overlap, and the proposed SAT-ASR system generates two sequences: Result-1 and Result-2. The word sequences were the token level text generated by a Transformer-based language model [43].

	Token Level Text
Label-A	SHE WAS YOUNG UN TRI ED NERVOUS IT WAS A VISION OF SERIOUS DUTIES AND LITTLE COMPANY OF REALLY GREAT LO NE LINESS
Result-1	SHE WAS YOUNG UN TRI ED NERVOUS IT WAS A VISION OF SERIOUS DUTIES AND LITTLE COMPANY BUT REALLY GREAT LO NE LINESS <eos>
Label-B	HE HOPED THERE WOULD BE STE W FOR DINNER TURN IP S AND CARR OT S AND B RU IS ED POT A TO ES AND FAT MU T TON PIECES TO BE LAD LED OUT IN THICK PEPPER ED FLOUR FAT TEN ED SAUCE
Result-2	HE HOPED THERE WOULD BE STE W FOR DINNER TURN IP S AND CARR OT S AND B RU IS ED POT A TO ES AND FAT MU T TON PIECES TO BE LAD LED OUT IN THICK PEPPER ED FLOUR FAT AND SAUCE <eos>

shows an example output generated by the proposed SAT-ASR with a profile. The input was the mixed speech of one speaker saying Label-A and the other saying Label-B, and Result-1 and Result-2 were the output word sequences. The word sequences were generated using the token level text vocabulary of a Transformer-based language model [43], so there were several non-language tokens such as <eos>, meaning end of sentence. When comparing Label-A and Result-1, only a single error was observed: OF → BUT. When comparing Label-B and Result-2, only a single error was also observed: TEN → AND. In summary, the experimental results show that the proposed speaker-attributed training was highly effective in multi-speaker speech recognition tasks without adding significant storage and computational overheads.

5. Conclusions

This paper presents novel solutions to improve speech recognition performance in multi-speaker speech environments. In such settings, where multiple speeches occur simultaneously, traditional speech recognition systems are hard to distinguish between different speakers. In scenarios with overlapping speeches, we introduced the speaker-attributed training automatic speech recognition (SAT-ASR) model, featuring a novel structure and incorporating additional inputs such as speaker embeddings and a speaker profile method. We proposed using i-, d-, and x-vectors as effective cues to split the multiple-speaker information mixed in the input features. We also proposed utilizing a profile matrix constructed by the embedding vectors of the enrolled speakers. Because speaker embedding vectors are extracted from mixed speech, they are regularized by being replaced with the linearly weighted combinations of the enrolled embedding vectors. The effectiveness of the proposed method was verified by multi-speaker speech recognition experiments on LibriMix datasets. A comparative analysis of using the speech separation implemented by DPRNN-TasNet as inputs to the single-speaker ASR model, conventional multi-speaker ASR models implemented by Transformer networks, and the proposed SAT-ASR models were carried out. The word error rates (WERs) measured on a designated test set of LibriMix were 13.6%, 8.43%, and 8.1%, respectively. When employing speaker profiles, the SAT-ASR model further improved its WER to 7.48%, surpassing all the other multi-speaker ASR solutions. The computational efficiencies were also compared in terms of the number of trainable parameters and training times. The proposed methods required 336 and 328 GPU hours with and without speaker profiles, while the baseline multi-speaker model required 320 GPU hours, resulting in up to 5% additional training time.

One of the major advantages of the proposed methods was that the pretrained components of the single-speaker ASR models can be reused without fine tuning by multi-speaker utterances. Those parts were speech recognition encoders yielding text outputs. Training a single-speaker ASR model from scratch required 892 h, but it was not required to be updated again. These methods not only improved the accuracy of the speech recognition systems, but they were also adaptable to a variety of complex speech scenarios with a small amount of computational overhead.

Future research will focus on expanding the SAT-ASR model to be applicable to scenarios with an arbitrary number of speakers. To achieve this, we will explore techniques for automatically detecting the number of speakers in an audio input and adjusting the encoder of the proposed model accordingly. Additionally, we aim to improve the model’s structure to ensure that, when recognizing a single-speaker speech in multi-speaker environments, the computational load is comparable to that of conventional single-speaker ASR models. Through these plans, we intend to achieve more accurate multi-speaker speech recognition in complex real-world environments.