Improving Transformer Based End-to-End Code-Switching Speech Recognition Using Language Identification

Abstract

A Recurrent Neural Networks (RNN) based attention model has been used in code-switching speech recognition (CSSR). However, due to the sequential computation constraint of RNN, there are stronger short-range dependencies and weaker long-range dependencies, which makes it hard to immediately switch languages in CSSR. Firstly, to deal with this problem, we introduce the CTC-Transformer, relying entirely on a self-attention mechanism to draw global dependencies and adopting connectionist temporal classification (CTC) as an auxiliary task for better convergence. Secondly, we proposed two multi-task learning recipes, where a language identification (LID) auxiliary task is learned in addition to the CTC-Transformer automatic speech recognition (ASR) task. Thirdly, we study a decoding strategy to combine the LID into an ASR task. Experiments on the SEAME corpus demonstrate the effects of the proposed methods, achieving a mixed error rate (MER) of $30.95\%$. It obtains up to $19.35\%$ relative MER reduction compared to the baseline RNN-based CTC-Attention system, and $8.86\%$ relative MER reduction compared to the baseline CTC-Transformer system.

1. Introduction

Code-switching (CS) speech is defined as speech which contains more than one language within an utterance [1]. With the development of globalization, this multilingual phenomenon has become increasingly common in real life, so the research on this phenomenon has attracted growing attention. Traditionally, research on a Gaussian mixture model based hidden Markov models (GMM-HMM) and deep neural network based hidden Markov model (DNN-HMM) framework for code-switching speech recognition (CSSR) [2,3] focuses on two challenges: lack of language model training data at CS points, co-articulation effects between phonemes at CS points. Therefore, it is difficult to reliably estimate the probability of word sequences where CS appears and to model the phonemes at CS points. To address the former challenge, statistical machine translation is utilized to manufacture artificial CS training text [4]. Several methods are proposed to improve the performance of language modeling to CS speech: recurrent neural network language models and factored language models with the integration of part-of-speech tag, language information, or syntactic and semantic features [5,6,7]. To address the latter challenge, speaker adaptation, phone sharing and phone merging are applied [4]. Recently, an End-to-End (E2E) approach for the CSSR task has attracted increasing interest [8,9,10,11]. By predicting graphemes or characters directly from acoustic information without predefined alignment, the E2E system can considerably reduce the effort of building automatic speech recognition (ASR) systems. In the mean time, the need of expert linguistic knowledge is also eliminated, which makes it an attractive choice for CSSR. Previous works mainly adopted two types of E2E methods in the CSSR task: connectionist temporal classification (CTC) [12] and the RNN-based attention method [13,14,15]. The CTC objective function simplifies acoustic modeling into learning a RNN over pairs of speech and context-independent (CI) label sequences, without requiring a frame-level alignment of the target labels for a training utterance [16]. The RNN-based attention method consists of an RNN encoder and an attention-based RNN decoder, which maps acoustic speech into a high-level representation and recognizes symbols conditioned on previous predictions, respectively [13,14,15]. A joint CTC-Attention multi-task learning model is presented to combine the benefit of both types of systems [17,18]. However, RNN remains as the sequential computation constraint. Therefore, stronger short-range dependencies and weaker long-range dependencies exist in encoder outputs and decoder outputs, which makes it hard to immediately switch languages in CSSR. Recently, the Transformer [19,20] has achieved state-of-the-art performances in many monolingual ASRs [21]. It transduces sequential data with its self-attention mechanism, which replaces the RNN in previous works [15,18]. Since self-attention mechanisms utilize global context—that is, all frames learn time dependency inside the input sequence to achieve sequence transduction in parallel—information transmission is the same for each location to draw global dependencies, which makes it possible to switch more freely at CS points. Therefore, in this paper, we apply a joint CTC-Transformer framework for CSSR. Then, we study different multi-task learning recipes, where a language identification (LID) auxiliary task is learned in addition to the ASR task. Lastly, we study a decoding strategy to combine the LID information into ASR. All of our experiments are conducted on the SEAME corpus.

The paper is organized as follows. Related works are presented in Section 2. The multi-task learning recipes and LID joint decoding are studied in Section 3. Experimental setups and results analysis are described in Section 4. Some conclusions are drawn in Section 5.

2. Related Work

2.1. Transformer Based E2E Architecture

The Transformer contains an Encoder network and a Decoder network [21,22,23]. Both the Encoder and the Decoder consist of several layers stacked, as shown in Figure 1a,b, respectively. The Encoder transforms the input features $X=[x_1,\dots ,x_T]$ into a sequence of encoded features $H_e=[h_{e,1},\dots ,h_{e,T}]$, as follows:

$$\begin{array}{cc}\hfill & H_0=CNN\left(X\right)+PE\hfill \\ \hfill & H_i^{{}^{\prime }}=H_i+MH{A}_i(N\left(H_i\right),N\left(H_i\right),N\left(H_i\right))\hfill \\ \hfill & H_{i+1}=H_i^{{}^{\prime }}+F{F}_i\left(N\left(H_i^{{}^{\prime }}\right)\right),\hfill \end{array}$$

(1)

where $i=0,\dots ,e-1$, e is the number of encoder layers, $CNN(·)$ is a convolution network, $PE$ is positional encoding, $MHA(·)$ is a multi-head self-attention mechanism, and $F{F}_i$ is a positionwise fully connected feed-forward network. In this work, layer normalization $N(·)$ is employed before each sub-layer. The Decoder receives the encoded features $H_e$ and the label sequence $Y[0:l-1]$ to emit the probabilities of the Decoder output units set $\mathcal{Y}\left[l\right]=[y_l^1,\dots ,y_l^{d^{units}}]$ of the l-th step, and then determine the subsequent $[{\widehat{y}}_1,\dots ,{\widehat{y}}_l]$, as Equations (2)–(4):

$$E_{dec}=Embed\left(Y[0:l-1]\right).$$

(2)

$Embed(·)$ is an embedding layer that transforms a sequence of labels $Y[0:l-1]$ into a sequence of learnable vectors $E_{dec}\in {\mathbb{R}}^{l\times d^{att}}$, and $d^{att}$ is the dimension of attentions.

$$\begin{array}{cc}\hfill & D_0=E_{dec}+PE\hfill \\ \hfill & D_j^{{}^{\prime }}=D_j+MH{A}_j^{self}(N\left(D_j\right),N\left(D_j\right),N\left(D_j\right))\hfill \\ \hfill & D_j^{{}^{\prime \prime }}=D_j^{{}^{\prime }}+MH{A}_j^{src}(N\left(D_j^{{}^{\prime }}\right),H_e,H_e)\hfill \\ \hfill & D_{j+1}=D_j^{{}^{″}}+F{F}_j\left(N\left(D_j^{{}^{″}}\right)\right),\hfill \end{array}$$

(3)

where $j=0,\dots ,d-1$, d is the number of decoder layers.

$$\begin{array}{cc}\hfill & [p_{att}\left(\mathcal{Y}\left[1\right]\right),\dots ,p_{att}\left(\mathcal{Y}\left[l\right]\right)]=softmax\left(D_d{W}^o\right)\hfill \\ \hfill & {\widehat{y}}_{l^{{}^{\prime }}}=\underset{y_{l^{{}^{\prime }}}\in \mathcal{Y}}{argmax}{p}_{att}\left(\mathcal{Y}\left[l^{{}^{\prime }}\right]\right)l^{{}^{\prime }}\in 1,\dots ,l\hfill \end{array}$$

(4)

where learnable weight matrices $W^o\in {\mathbb{R}}^{d^{att}\times d^{units}}$ belong to the output linear layer, and $d^{units}$ is the number of output units.

In this work, in the training stage, ground truth sequence ($Y[0:l-1]=[y_0,\dots ,y_{l-1}]$) is adopted as the input of the embedding layer, while in the decoding stage, predicted sequence ($Y[0:l-1]=[{\widehat{y}}_0,\dots ,{\widehat{y}}_{l-1}]$) is adopted.

2.2. Self-Attention

Scaled Dot-Product Attention is commonly used as an attention function for the self-attention mechanism [21]. The input consists of queries(Q) and keys(K) of dimension $d_k$, and values(V) of dimension $d_v$. Scaled Dot-Product Attention is computed as Equation (5):

$$Att(Q,K,V)=softmax\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V.$$

(5)

To allow the model to jointly attend to information from different representation subspaces at different positions, [21] extends Equation (5) to multi-head attention Equation (6):

$$\begin{array}{cc}\hfill & MHA(Q,K,V)=Concat(h{d}_1,\dots ,h{d}_h)W\hfill \\ \hfill & h{d}_i=Att(Q{W}_i^Q,K{W}_i^K,V{W}_i^V),\hfill \end{array}$$

(6)

where h is the number of attention heads, $W_i^Q,W_i^K\in {\mathbb{R}}^{d_{att}\times d_k},W_i^V\in {\mathbb{R}}^{d_{att}\times d_v}$ and $W\in {\mathbb{R}}^{(h\times d_v)\times d_{att}}$ are learnable weight matrices.

3. Methods

3.1. CTC-Transformer Based CSSR Baseline System

We adopted a Transformer framework to build the CSSR system. However, the Transformer takes many more epochs for the monolingual ASR task to converge, let alone the CSSR task, which has many more model units. Inspired by [20], we added a CTC objective function to train the encoder of the Transformer. CTC helps the Transformer to converge with the forward–backward algorithm, enforcing a monotonic alignment between input features and output labels. The architecture of the CTC-Transformer baseline system is indicated in Figure 2.

Specifically, let X be the input acoustic sequence, Y be the output label sequence comprising Mandarin modeling units or English modeling units, let $L_{ctc}\left(Y\right|X)$ be the CTC objective loss [16] and $L_{att}\left(Y\right|X)$ be the attention-based objective loss. The $L_{att}\left(Y\right|X)$ loss is the cross entropy of predicted $\widehat{Y}$ and ground truth Y. The combination of $L_{ctc}\left(Y\right|X)$ and $L_{att}\left(Y\right|X)$ is adopted for the ASR task:

$$L_{ASR}\left(Y\right|X)=(1-\alpha )L_{att}\left(Y\right|X)+\alpha L_{ctc}\left(Y\right|X),$$

(7)

where $\alpha$ is a hyperparameter.

We chose a Chinese character as the Mandarin acoustic modeling unit, as it is the most common choice for E2E Mandarin ASR and it has shown a state-of-the-art performance in Mandarin ASR [24,25]. As for English, we chose the subword as the English unit. We adopted Byte Pair Encoding (BPE) [26] to generate subword units.

3.2. CSSR Multi-Task Learning with LID

In CSSR, modeling units belonging to different languages but with similar pronunciation are easy to confuse. Meanwhile, the language information was not used explicitly during training. LID is a process by which a computer analyzes and processes speech to determine which language it belongs to. So, we believe that adopting LID prediction as an auxiliary task can improve the CSSR performance. We sent the feature output from the encoder to the decoder, and the decoder output to the LID sequence corresponding to the feature sequence. The LID task and the ASR task share the same encoder, therefore we call it multitask learning. The objective loss $L_{LID}\left(Z\right|X)$ was added to extend the multi-task learning (MTL) objective loss:

$$L_{MTL}=(1-\beta )L_{ASR}\left(Y\right|X)+\beta L_{LID}\left(Z\right|X).$$

(8)

The $L_{LID}\left(Z\right|X)$ loss is the cross entropy of the predicted LID label sequence $\widehat{Z}$ and the ground truth LID label sequence Z. In this work, each LID label $z_l$ corresponds to an ASR label $y_l$. So the length of the LID label sequence Z is the same as that of the ASR label sequence Y. An example is shown in Figure 3.

We used label ‘E’ for English, and label ‘M’ for Mandarin. We used label ‘N’ for nonverbal, such as noise, laugh and so on, but we do not mention ‘N’ below, as it is not important in this study. Because the length of the Z sequence is the same as that of the Y sequence, and is inspired by the decoding method of the Y sequence, we used a similar structure to predict the Z sequence. In the training stage, the Decoder of the ASR task used $H_e$ and the information of the label sequence $(y_0,\dots ,y_{l-1})$ to predict $(y_1,\dots ,y_l)$. We studied what label sequence should participate to predict the next LID label. Specifically, we propose two strategies to implement the Z label prediction task.

• LID label sequence (LLS): as indicated in Figure 4a, the LID predictor receives the encoded features $H_e$ and the LID label sequence $(z_0,\dots ,z_{l-1})$ to predict $(z_1,\dots ,z_l)$. The LID predictor does not share the embedding layer with ASR predictor, and it has its own embedding layer:

  $$E_Z=Embe{d}_{LID}\left(Z[0:l-1]\right).$$

  (9)
  $Embe{d}_{LID}(·)$ transforms a sequence of labels $Z[0:l-1]$ into a sequence of learnable vectors $E_Z\in {\mathbb{R}}^{l\times d^{att}}$.
• ASR label sequence (ALS): just like LLS, except that the LID predictor does not receive the LID label sequence, but the ASR label sequence $(y_0,\dots ,y_{l-1})$, and in fact, the LID predictor shares the embedding layer with the ASR predictor. ALS can be implemented in two structures, one is the LID task sharing the decoder with the ASR task (ALS-share), the other is not sharing (ALS-indep), as indicated in Figure 4b,c, respectively.

On this basis, inspired by [20], we also operate another set of experiments adding joint CTC training for the LID task.

3.3. CSSR Joint Decoding with LID

There are similar pronunciation units across two languages; therefore, units of one language may be incorrectly identified as units with similar pronunciation to that of another language. Figure 5 shows an example in our experiment.

This problem may be related to the fact that LID is not used in decoding. As indicated in Figure 6, we integrate LID into the decoding process by conditionally modifying the ASR output probabilities $p_{att}\left(y_l^k\right)$ with the LID output probabilities $p_{att}\left(z_l^{m_k}\right)$:

• Firstly, ASR branch decoding and LID branch decoding are carried out simultaneously to obtain the ASR label ${\widehat{y}}_l$ and the LID label ${\widehat{z}}_l$ of the l-th step;
• ${\widehat{y}}_l$ can be uniquely mapped to ${\widehat{z}}_l^{{}^{\prime }}$, since there is no intersection between the Chinese modeling unit set and the English modeling unit set;
• If ${\widehat{z}}_l^{{}^{\prime }}$ is not in {‘E’,‘M’} or ${\widehat{z}}_l$ is not in {‘E’,‘M’}, $p_{att}\left(y_l^k\right)$ will not be modified.
• If both ${\widehat{z}}_l^{{}^{\prime }}$ and ${\widehat{z}}_l$ are in {‘E’,‘M’}, and ${\widehat{z}}_l^{{}^{\prime }}$ is different from ${\widehat{z}}_l$, then, modification and normalization will be added to $p_{att}\left(y_l^k\right)$.

  $$\begin{array}{cc}\hfill & p_{att}^{{}^{\prime }}\left(y_l^k\right)=\frac{p_{att}\left(y_l^k\right)\times p_{att}\left(z_l^{m_k}\right)}{{\sum }_k{p}_{att}\left(y_l^k\right)p_{att}\left(z_l^{m_k}\right)}\hfill \\ \hfill & p_{att}\left(y_l^k\right)=p_{att}^{{}^{\prime }}\left(y_l^k\right),\hfill \end{array}$$

  (10)

  where $k=1,\dots ,d^{units}$, $d^{units}$ is the number of ASR decoder output units, and $y_l^k$ is the k-th output unit of l-th step, and $m_k$ is the k-th ASR task output units mapping to the LID task output units; therefore, $z_l^{m_k}$ is the $m_k$-th output unit of l-th step of the LID task, $p_{att}\left(y_l^k\right)$ is the probability of the k-thASR output unit of the l-th step, $p_{att}\left(z_l^{m_k}\right)$ is the probability of the $m_k$-th LID output unit of the l-th step.

If $z_l^{m_k}$ is different to the corresponding language of $y_l^k$, then, after normalization, the value of $p_{att}\left(y_l^k\right)$ will decrease, which can reduce the probability of selecting the current error unit.

4. Experiments and Results

4.1. Data

We conduct experiments on the SEAME(South East Asia Mandarin–English) corpus [27], which was developed for spontaneous Mandarin–English code-switching research. We divide the SEAME corpus into three sets (train, development and test) by proportionally sampling speakers. The detailed statistics of the corpus division are presented in

Table 1. Statistics of the SEAME corpus.

	Train Set	Dev Set	Test Set	Total
speakers	145	2	8	155
Duration (h)	107	1.86	6.38	115.23
utterances	100802	1596	6276	108674

.

4.2. Baseline Setup

Firstly, we replicate two other baseline systems based on different frameworks—GMM-HMM [4] and RNN-based CTC-Attention [17]. Since the partition of training/development/test sets is not identical, there is a slight gap between our results and the original, but within the allowable range. A “big model” is commonly suggested for the Transformer [19,20] in monolingual ASR, but we find it unsuitable for CSSR due to insufficient CS data obtained and CSSR output units that are too large. In this work, we chose a “smaller model" for the Transformer($d^{att}=256$, $e=6$, $d=3$). The input speech is represented as a sequence of a 40-dim filterbank feature. The filterbank feature is firstly subsampled by a two layer time-axis convolutional neural network with $ReLU$ activation (stride size is 2, kernel size is 3, the number of channels is 256). The loss weight $\alpha$ for the CTC joint training is set to $0.3$. To prevent training from overfitting to the training set, label smoothing [28] with a penalty of 0.1 is applied. For Mandarin modeling units, a set of 2639 Chinese characters is used, covering the Chinese characters that appear in the training text. For English, a set of 200 subwords is used, which is trained on English segments of the training set using the BPE method [26].

As shown in

Table 2. MER(%) of different framework CSSR baseline systems.

Baseline System	MER(%)
GMM-HMM	39.6
RNN-based CTC-Attention	38.38
CTC-Transformer	33.96

, the CTC-Transformer baseline had a mixed error rate (MER) of $33.96\%$, better than that of the RNN-based CTC-Attention baseline ($38.38\%$) and the GMM-HMM baseline ($39.6\%$).

4.3. Effects of Different MTL Strategies for Combination of LID and ASR

We conducted experiments using a different choice of MTL weight $\beta$, and we chose the best weight $\beta =0.1$ in the following experiments. As shown in

Table 3. MER(%) of ASR task corresponding to three MTL implementations: LLS, ALS-share and ALS-indep.

MTL Strategy	LID-CTC	MER(%)
no LID task	no	33.96
LLS	no	32.24
	yes	31.37
ALS-share	no	32.79
	yes	31.84
ALS-indep	no	34.4

, LLS and ALS-share achieve an equivalent effect, both better than the CTC-Transformer baseline, consistent with our expectations. However, the effect of ALS-indep is worse, therefore we did not use ALS-indep for subsequent experiments.

To add joint CTC training for LID, the LID CTC weight is set to 0.3, the same as the ASR CTC weight ($\alpha$ = 0.3). As shown in Table 3, the LID auxiliary task with joint CTC training can better assist the ASR main task, because CTC can learn to align the speech feature and the LID label sequence explicitly.

4.4. Effects of Joint Decoding with LID

As shown in

Table 4. MER(%) of ASR task (using CTC-Transformer architecture), corresponding to joint decoding with LID or not.

MTL Strategy	LID-CTC	LID Joint Decoding	MER(%)
LLS	yes	no	31.37
LLS	yes	yes	30.95
ALS-share	yes	no	31.84
ALS-share	yes	yes	31.84

, joint LID decoding improves LLS, but has no effect on ALS-share. For ALS-share, the LID task and the ASR task use the same decoder, which makes the result of the LID task more closely related to that of the ASR task. In contrast, as for LLS, the LID task is relatively independent of the ASR task. Therefore, LID decoder training in LLS is more capable of correcting language errors in the ASR task. Under the configuration of the LLS method, CTC joint training for the LID task and LID joint decoding, the final system achieves an MER of (30.95%), obtaining up to a (19.35%) and (8.86%) relative MER reduction compared to the RNN-based CTC-Attention baseline system (38.38%) and the CTC-Transformer baseline system (33.96%) respectively.

5. Conclusions

In this work, we introduce a CTC-Transformer based E2E model for Mandarin–English CSSR, which outperforms most of the traditional systems on the SEAME corpus. As for the inclusion of LID, we propose two LID multi-task learning strategies: LLS and ALS (ALS-share and ALS-indep). LLS and ALS-share have a comparable promotion effect. Furthermore, we study a decoding strategy to combine the LID information into the ASR task and it slightly improves the performance in the case of LLS. The final system with the proposed methods achieved an MER of $30.95\%$, obtaining up to a $19.35\%$ and $8.86\%$ relative MER reduction compared to the RNN-based CTC-Attention baseline system ($38.38\%$) and the CTC-Transformer baseline system ($33.96\%$), respectively.