Threshold-Based Combination of Ideal Binary Mask and Ideal Ratio Mask for Single-Channel Speech Separation

Abstract

An effective approach to addressing the speech separation problem is utilizing a time–frequency (T-F) mask. The ideal binary mask (IBM) and ideal ratio mask (IRM) have long been widely used to separate speech signals. However, the IBM is better at improving speech intelligibility, while the IRM is better at improving speech quality. To leverage their respective strengths and overcome weaknesses, we propose an ideal threshold-based mask (ITM) to combine these two masks. By adjusting two thresholds, these two masks are combined to jointly act on speech separation. We list the impact of using different threshold combinations on speech separation performance under ideal conditions and discuss a reasonable range for fine tuning the thresholds. By using masks as a training target, to evaluate the effectiveness of the proposed method, we conducted supervised speech separation experiments applying a deep neural network (DNN) and long short-term memory (LSTM), the results of which were measured by three objective indicators: the signal-to-distortion ratio (SDR), signal-to-interference ratio (SIR), and signal-to-artifact ratio improvement (SAR). Experimental results show that the proposed mask combines the strengths of the IBM and IRM and implies that the accuracy of speech separation can potentially be further improved by effectively leveraging the advantages of different masks.

1. Introduction

Speech separation [1,2,3,4] aims to extract the target speech signal from the noisy speech signal that may contain not only the background noise but also the non-target speech signals. This method can be adopted for speech recognition [5,6], mobile communication [7,8], speaker recognition [9,10], and so forth. It is classified as single-channel speech separation with only one microphone [11,12] and as multi-channel speech separation with multiple microphones [13,14]. Single-channel speech separation is a more challenging task than multi-channel speech separation as it cannot leverage spatial information, such as different arrival times and arrival directions of speech signals, when the target and interfering sound sources are co-located or near to each other. The single-channel speech separation discussed in this paper refers to the separation of speech from multiple speakers.

Over the past few decades, several approaches for single-channel speech separation have been proposed, including speech enhancement techniques [15,16] (among which the most widely used is spectral subtraction [17]) and model-based (such as hidden Markov model (HMM) [18] and Gaussian mixture model (GMM) [19]) techniques. To use these methods, some prior knowledge about the speech or background interference is required, and the speech intelligibility is difficult to improve by using these methods. Recently, computational auditory scene analysis (CASA) [20,21,22] has been proposed, which uses a set of perceptual cues such as pitch, voice onset/offset, temporal continuity, harmonic structure, modulation correlation, and so forth, to separate the desired speech by masking the interfering sources.

In CASA systems, time–frequency masks are typically used to retain the target speech signal as much as possible while removing interferences. The ideal binary mask (IBM) [23,24] extracts the dominant components of the target speech in the mixed speech signal by being assigned a value of 1 and directly discards the non-dominant parts by being assigned a value of 0. In contrast, the ideal ratio mask (IRM) [25,26] attempts to preserve as much of the target speech signal components as possible by being assigned any value within the range of 0 to 1. These two types of masks are widely used and exhibit different performances under various speech separation conditions. Generally, the IBM is more conducive to improving speech intelligibility [27,28], while the IRM is more conducive to improving speech quality [29,30]. It is feasible to establish a combined mask using their respective advantages to improve the performance of speech separation.

In this paper, we propose a method using a threshold-based combination mask for single-channel speech separation. Taking into account the differences in mixed speech from different speakers, the mask with thresholds is used to determine the critical point of the strength between the target and interfering speech signals, thus enhancing the performance of speech separation by combining the advantages of the IBM and IRM. With the development of deep learning (DL), masks have been used as a training target in speech separation, achieving good performance [3,29,30,31]. We conducted speech separation experiments under both ideal conditions and DL models to evaluate the effectiveness of our proposed method.

This paper is organized as follows. In Section 2, the notation and formulation of this research are explained, which contain notation and background information of the IBM and IRM. In Section 3, the single-channel speech separation using the threshold-based combination mask is proposed. Then, evaluation experiments are described in Section 4. Finally, the conclusion is given with a discussion of future work in Section 5.

2. Ideal Binary Mask (IBM) and Ideal Ratio Mask (IRM)

In this section, we first introduce the relevant notation and formulation of using masks for speech separation. Then, we review the two commonly used masks, the IBM and IRM, and discuss their respective advantages and disadvantages.

2.1. Notation and Formulation

Single-channel speech separation is commonly formulated as an estimation problem based on the following:

$$x\left(n\right)=\sum _{i=1}^C{x}_i\left(n\right),$$

(1)

where $x\left(n\right)$ is the acquired signal (mixed signal), n is the time index, and $x_i\left(n\right)$ is the clean speech signals, for the index of the speaker i = 1, 2, …, C. To simplify the discussion, we consider C = 2, in which two speakers speak at the same time.

The short-time Fourier transform (STFT) of $x\left(n\right)$ is represented as

$$X(k,l)=\sum _{n=0}^{N-1}x(n+lH)w\left(n\right)e^{-j2\pi kn/N},$$

(2)

where $w\left(n\right)$ is the window function, k, l, H, j, and N are the frequency bin index, frame index, window shift, the imaginary unit, and window length, respectively.

To separate the mixed speech signal into each individual speech signal, we consider one speech signal as the target speech signal (e.g., speech signal $x_1\left(n\right)$). To obtain the target speech signal, the T-F mask is used in this study. As mentioned in Section 1, the IBM and IRM are widely used. The IBM (also known as the hard mask) of target speech signal is defined by

$$M_t^{\mathrm{B}}(k,l)=\left\{\begin{array}{cc}\hfill 1,\hfill & |X_t(k,l)|\ge |X_{i\ne t}(k,l)|\hfill \\ \hfill 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.,$$

(3)

where $X_t(k,l)$ denotes the STFT of $x_t\left(n\right)$, $|X_t(k,l)|$ denotes the amplitude spectrum of $X_t(k,l)$, and t denotes the index of the target speaker. The IRM (also known as the soft mask) of target speech signal is defined by

$$M_t^{\mathrm{R}}(k,l)={\displaystyle \frac{|X_t(k,l)|}{{\sum }_{i=1}^C\left|X_i(k,l)\right|}}.$$

(4)

By using the T-F mask, the target speech signal $x_1\left(n\right)$ (as mentioned earlier) can be extracted from the mixed speech as follows:

$${\widehat{X}}_1(k,l)=M_t(k,l)\odot X(k,l).$$

(5)

Accordingly, the other speech signal $x_2\left(n\right)$ (it is considered as the non-target signal) can be represented as

$${\widehat{X}}_2(k,l)=(1-M_t(k,l))\odot X(k,l),$$

(6)

where $\widehat{X}(k,l)$ is the estimated spectrum of $X(k,l)$, $M_t(k,l)$ is the T-F mask for $X_t(k,l)$, and ⊙ is the Hadamard product. As shown in Equations (3) and (4), the IBM and IRM have different characteristics, and the spectra estimated by them are also different. Finally, after separating the speech signal by Equations (5) and (6), the time-domain signal of $\widehat{X}(k,l)$ is obtained by inverse STFT.

2.2. Advantages and Disadvantages of IBM and IRM

As described above, the IBM is a hard mask, which can take two values, 0 and 1. It only preserves the dominant signal in the mixed signal, as shown in Figure 1a,c. This means that using this mask is beneficial for separating each time–frequency component from the mixed signal with significant differences in signal strength, but it is not effective at separating each time–frequency component from the mixed signal with comparable strength. In other words, the target signal is difficult to separate from the mixed signal when the signal powers of both signals are almost the same, as shown in Figure 1b.

Regarding the IRM, as a soft mask, it can have any value between 0 and 1. Based on the above definition and description, it can preserve all the time–frequency components of the target signal as much as possible, regardless of whether the target signal dominates in these regions or not. This indiscriminate preservation can lead to more residual interfering signals because such preservation is very redundant in mixed signals where one side absolutely dominates, as shown in Figure 1a,c.

In summary, the IBM is effective for removing interference signals but not for recovering the target signal, while the IRM is advantageous for preserving the target signal but not for removing the interference signals. Therefore, the extent to which interference signals are retained is a concern in the method of speech separation by using masks, which also greatly affects the performance of extracted signals. It is necessary to explore a new method to combine the advantages of the above two masks to control the retention of interference signals, thereby improving the accuracy of speech separation.

3. Proposed T-F Mask for Speech Separation

In this section, we present the proposed combination mask and describe the use of masks as training targets for speech separation. Figure 2 shows an overview of the proposed method by using the T-F mask and DL-based model.

3.1. Threshold-Based Combination Mask

On the basis of the discussion and explanation in Section 2.2, we consider the different intensities of the target signal and the interference signal in the mixed signal. As shown in Figure 1a,c, when one signal dominates, the use of the IBM is entirely feasible. This implies that we can directly discard the signal with very weak intensity. However, when the target signal is comparable to the interference signal, as shown in Figure 1b, we cannot directly discard either of them. Therefore, to improve the performance of speech separation, the IRM needs to be used and the energy of both signals needs to be fully preserved.

Based on the above analysis, considering the differing strengths and combinations of the target signal and the interference signal, it is highly reasonable to use a combination of these two masks for speech separation. It is expected that when one signal dominates in strength, the IBM can be used to effectively discard the interference signal, and when the strengths of both signals are comparable, the IRM can be used to fully preserve the desired target signal. Here, we use thresholds to determine the differences in signal strength and adopt different masks on the basis of these differences. By determining the upper and lower limit values, the target signal can be retained as much as possible while also more effectively discarding the interference signal. Here, we define an ideal threshold-based mask (ITM), shown as

$$M_t^{\mathrm{T}}(k,l)=\left\{\begin{array}{cc}1,\hfill & M_t^{\mathrm{R}}(k,l)\ge \alpha \hfill \\ 0,\hfill & M_t^{\mathrm{R}}(k,l)<\beta \hfill \\ M_t^{\mathrm{R}}(k,l),\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(7)

where $\alpha$ and $\beta$ are thresholds applied to all elements, and these thresholds are used to compare with the ideal ratio mask to determine which mask (IBM or IRM) should be used for the ITM. For regions where the target and interfering signals significantly differ in strength, the upper threshold $\alpha$ should be used to enhance the target signal, and the lower threshold $\beta$ should be used to further discard the interference signal. For regions where there is a small difference in strength between the target and interfering signals, the ratio mask is adopted to effectively preserve the energy of both signals for speech separation.

3.2. Using Masks as Training Targets

Deep neural network (DNN) [32,33,34] and long short-term memory (LSTM) [35,36,37] are popular DL networks that effectively generate the estimated mask. Supervised speech separation is divided into two stages, the training stage and the separation stage, which are shown in Figure 2. In the training stage, we use the STFT to obtain the features of the speech signals and calculate the ideal mask, in which the DL model is trained for generating the estimated T-F mask of the target speaker. In the separation stage, we extract the spectrum of the target speech from the mixed speech utilizing the DL model, which has already been trained in the previous training stage. Then, the spectrum of the target speech signal is converted into the time waveform by inverse STFT.

Figure 2. Diagram of deep learning-based speech separation system using mask as training target.

Figure 2. Diagram of deep learning-based speech separation system using mask as training target.

In this paper, the DL network is designed to estimate the mask calculated by Equation (7). The speech features are the amplitude spectra of the speech signal. We compute the masks for the target speech signal in advance, which are used as the training targets for the network. The mask is estimated to be equal to the ideal mask. The network is trained using mean-square error as the loss function. The mean-square error of target speech $E_t$ is represented by

$$E_t={\displaystyle \frac{1}{KL}}\sum _{k=0}^{K-1}\sum _{l=0}^{L-1}{\left({\widehat{M}}_t(k,l)-M_t(k,l)\right)}^2,$$

(8)

where $M_t(k,l)$ is the ideal T-F mask (either IBM, IRM, or ITM) for the target speaker, ${\widehat{M}}_t(k,l)$ is the estimated T-F mask for the target speaker (which is also the output of the separation model), K is the number of frequency bins, and L is the number of frames. In addition, because the hard mask is not smooth, we use the soft mask as the training target of the network and then generate a hard mask through discrimination, with values greater than or equal to 0.5 being set to 1, and others set to 0.

4. Experiments and Results

We conducted experiments to evaluate the effectiveness of the proposed method. The experiments can be divided into two parts. In the first part, the performances of the conventional and proposed masks were evaluated in an ideal condition, i.e., the masks are directly calculated from the target and interference signals. In the second part, the performances were evaluated with the masks estimated using the DL model.

4.1. Experimental Setup

All experiments were implemented by using Matlab R2022b. The experimental conditions are shown in

Table 1. Experimental conditions.

Data	TIMIT Acoustic Phonetic Continuous Speech Corpus
Sampling frequency	16 kHz
Frame length	32 ms
Frame overlap	75%
Window	Hann

.

4.1.1. Dataset

We conducted the speech separation experiments to justify the effectiveness of our proposed methods. In the experiments, three male and three female speakers were chosen from the TIMIT Acoustic Phonetic Continuous Speech Corpus [38]. The speech signals from 2 speakers were mixed to create 15 different sets of two-speaker recordings: male+male, male+female, and female+female. For every speaker, nine TIMIT sentences, respectively, were used for training, and the remaining sentence was used as the test set [39]. All the mixed speech database was created at the ratio of $-5$, 0, and 5 dB [40]. The whole database in this evaluation experiment is more than three hours.

4.1.2. Experimental Configurations

We use STFT (the frame length was set to 512 samples) to obtain 257-dimensional spectral features for training DL models. The parameters of STFT are shown in Table 1. As a DL model, DNN and LSTM were used for conducting the speech separation experiments, respectively. For the DNN model, a feed-forward neural network is applied. The input and output sizes were set to 257 × 10 units, which denoted input and output layers with a 10-frame context. We employed the sigmoid activation function for the network. There are three layers of the network. The dropout layer is used for training to reduce overfitting with a loss rate of 10%. The adaptive moment estimation (ADAM) [41] optimizer is used to train DNN. For the LSTM model, the number of layers in the network is two. The dropout layer is used for training to reduce overfitting with a loss rate of 20%. The number of units is 640 in hidden layer 1 and 500 in hidden layer 2. ADAM is also used in the LSTM network.

The IBM and IRM in this experiment were calculated using Equations (3) and (4), respectively. The proposed mask (7) was calculated with the threshold $\alpha$ varying from 0.5 to 0.9 (in 0.1 increments), and the threshold $\beta$ varying from 0.1 to 0.5 (in 0.1 increments). As a result, 25 different combinations are formed. We obtained separated target and interference speech on the test set, evaluated the separation results with objective metrics, and took averages for comparing the speech separation performances.

4.1.3. Evaluation Metrics

Three indicators are used to evaluate the segregation performance: the signal-to-distortion ratio (SDR), signal-to-interference ratio (SIR), and signal-to-artifact ratio (SAR) [42]. The SDR is the ratio of the power of the target signal to the power of the difference between the target signal and the estimated signal. It reflects the overall separation performance in a speech separation task. The SIR reflects errors due to incomplete removal of the interfering signal, while the SAR reflects errors caused by additional artifacts introduced during the speech separation process. These metrics can be calculated [42] using the following formulas:

$$\mathrm{SDR}=10{\log }_{10}{\displaystyle \frac{\parallel {\mathbf{s}}_{\mathrm{target}}{\parallel }^2}{\parallel {\mathbf{e}}_{\mathrm{interf}}+{\mathbf{e}}_{\mathrm{artif}}{\parallel }^2}},$$

(9)

$$\mathrm{SIR}=10{\log }_{10}{\displaystyle \frac{\parallel {\mathbf{s}}_{\mathrm{target}}{\parallel }^2}{\parallel {\mathbf{e}}_{\mathrm{interf}}{\parallel }^2}},$$

(10)

$$\mathrm{SAR}=10{\log }_{10}{\displaystyle \frac{\parallel {\mathbf{s}}_{\mathrm{target}}+{\mathbf{e}}_{\mathrm{interf}}{\parallel }^2}{\parallel {\mathbf{e}}_{\mathrm{artif}}{\parallel }^2}},$$

(11)

where ${\mathbf{s}}_{\mathrm{target}}$ is the target speech signal, ${\mathbf{e}}_{\mathrm{interf}}$ is the interference speech signal, ${\mathbf{e}}_{\mathrm{artif}}$ is the artifact signal, and ${\parallel \mathbf{s}\parallel }^2$ (or ${\parallel \mathbf{e}\parallel }^2$) is the power of the signal. When evaluating separation performance, $x_1$ (or $y_1$) and $x_2$ (or $y_2$) are treated as the target speech signal and the interfering speech signal, respectively. All these signals are obtained by decomposing [42] the estimated signal. The higher these indicators, the higher the accuracy of speech separation.

4.2. Experimental Results and Discussions

4.2.1. Comparison of Proposed and Conventional Masks in Ideal Conditions

First, we compared the separation results by using various masks under ideal conditions (using the ideal masks).

Table 2. Evaluation metrics [dB] of different variants of ideal mask. (IBM: ideal binary mask, IRM: ideal ratio mask).

	SDR	SIR	SAR
IBM	11.53	28.36	11.73
IRM	$11.88$	19.58	$12.85$

lists the results of the speech separation experiments using the conventional method by the IBM and IRM. Table 2 shows that the IBM can better remove interference signals but leads to greater signal distortion. On the other hand, the IRM has a better degree of signal distortion but preserves more interference signals. From these results, the IBM has a better indicator of the SIR (8.77 dB higher than the IRM), and the IRM has a better indicator of the SDR (0.35 dB higher than the IRM). The IRM also improves the SAR more than the IBM does (1.12 dB higher).

Table 3. The ideal value of the SDR [dB] at different values of $\alpha$ and $\beta$.

$\mathit{\alpha }$	$\mathit{\beta }$
	0.1	0.2	0.3	0.4	0.5
0.5	11.60	11.87	11.88	11.86	11.53
0.6	11.92	12.18	12.24	12.05	11.87
0.7	12.10	12.35	12.40	12.16	11.99
0.8	12.05	12.27	12.35	12.13	11.95
0.9	11.89	12.13	12.20	11.97	11.87

,

Table 4. The ideal value of the SIR [dB] at different values of $\alpha$ and $\beta$.

$\mathit{\alpha }$	$\mathit{\beta }$
	0.1	0.2	0.3	0.4	0.5
0.5	26.11	26.92	27.63	28.13	28.36
0.6	24.63	25.53	26.42	27.30	28.17
0.7	23.35	24.30	25.29	26.38	27.66
0.8	22.31	23.28	24.32	25.53	27.02
0.9	21.41	22.40	23.46	24.73	26.32

and

Table 5. The ideal value of the SAR [dB] at different values of $\alpha$ and $\beta$.

$\mathit{\alpha }$	$\mathit{\beta }$
	0.1	0.2	0.3	0.4	0.5
0.5	12.06	12.24	12.16	12.10	11.73
0.6	12.39	12.55	12.54	12.30	12.11
0.7	12.61	12.77	12.74	12.46	12.28
0.8	12.63	12.75	12.76	12.50	12.33
0.9	12.56	12.70	12.71	12.45	12.36

show the results of the speech separation experiments using the proposed combination masks with various combinations of the thresholds. From Table 3, Table 4 and Table 5, the proposed combination mask shows the different results with each combination of the thresholds. As shown in Table 3, for the restoration of the target signal, better separation results can be achieved by adjusting the thresholds. Moreover, compared with both the IBM and IRM, using the ITM in certain threshold regions can improve the quality of target speech in speech separation. Table 3 shows that the best results in terms of the SDR are when $\alpha$ is in the range of [0.6, 0.8] and $\beta$ is in the range of [0.2, 0.4]. This can be expressed by the comparison results of the SDR: the average value of these results is 0.71 dB higher than that of the IBM and 0.36 dB higher than that of the IRM.

For the removal of interference signals, as shown in Table 4, when the threshold $\alpha$ is fixed, the value of the SIR increases with the increment of the threshold $\beta$. When the threshold $\beta$ is fixed, the value of the SIR generally shows a downward trend with the increment of the threshold $\alpha$. From the preliminary experiments, we found that, the smaller the difference between the thresholds $\alpha$ and $\beta$, the stronger the ability to remove interference signals. The best separation performance was achieved by ITM ($\alpha$ = 0.5 and $\beta$ = 0.5), the same as indicated by the results of the IBM.

Table 5 shows the results for the measurement of the degree of the SAR. These results are highly consistent with those indicated by the SDR. This means that the higher the SDR value, the higher the SAR value. Moreover, similar to the SDR, the best SAR results are also distributed in these regions, where $\alpha$ is in the range of [0.6, 0.8] and $\beta$ is in the range of [0.2, 0.4]. This can be expressed by the comparison results of the SAR: the average value of these results is 0.86 dB higher than that of the IBM but 0.26 dB lower than that of the IRM. The above results and analysis show that, with fine-tuned thresholds, the proposed threshold-based combination mask can achieve better speech separation results in terms of objective metrics (SDR, SIR, and SAR) under the ideal conditions. Especially when we set these two thresholds close to the middle value (in the setting of this paper, $\alpha$ = 0.7 and $\beta$ = 0.3), we can maximize the advantages of combining the IBM and IRM, thereby improving the performance of speech separation in this experiment.

4.2.2. Comparison of Proposed and Conventional Masks Based on Deep Learning Model for Speech Separation

We performed the speech separation experiments using different masks, and these experiments were based on the DNN and LSTM models. Here, on the basis of the previous discussion, in this experiment, we selected several representative masks (under different threshold combinations) to present and compare the results of these experiments. These results are presented in Figure 3, Figure 4 and Figure 5.

From Figure 3 and Figure 4, The IBM is superior to the IRM at removing interference signals, while the IRM is superior to the IBM at preserving the target signal. On the basis of different threshold combinations, the proposed ITM presented different separation results. Specifically, the ITM1 ($\alpha$ = 0.5, $\beta$ = 0.1) and ITM4 ($\alpha$ = 0.9, $\beta$ = 0.5) are closer to the IBM, while the ITM2 ($\alpha$ = 0.7, $\beta$ = 0.3) and ITM3 ($\alpha$ = 0.9, $\beta$ = 0.1) are closer to the IRM. The best overall result is presented by ITM2. When the DNN model is used, it improves the SDR by 1.28 dB, the SIR by −2.70 dB, and the SAR by 2.12 dB compared with the IBM and improves the SDR by 0.21 dB, the SIR by 3.69 dB, and the SAR by −1.65 dB compared with the IRM. When the LSTM model is used, it improves the SDR by 0.96 dB, the SIR by −2.88 dB, and the SAR by 1.77 dB compared with the IBM and improves the SDR by 0.29 dB, the SIR by 2.75 dB, and the SAR by −1.14 dB compared with the IRM. All of these results are consistent with what we achieved in the ideal conditions, as described in the previous sections.

To further validate these results, we compare the power spectrograms of the experimental results, which are shown in Figure 5. For example, at approximately 2 s into the speech signal of speaker 1, as well as 1.5 s into speech signal of speaker 2, we can observe the differences in separation performance presented under different masks. It shows that the IBM can more effectively remove interference speech signals, but compared with clean speech, the spectrum shows significant distortion. Also, the IRM can better preserve the original target speech signal, but compared with clean speech, the spectrum shows that more interference components are generated. ITM can generate fewer interference components while retaining the original target signal spectrum structure. From the spectrum graph, it can also be concluded that ITM combines the advantages of the IBM and IRM, overcoming their shortcomings, indicating that our proposed method is effective. These analyses show that the proposed method can improve the performance of speech separation.

5. Conclusions and Future Works

In this paper, we proposed an ideal threshold-based mask (ITM) combining the strengths of the ideal binary mask (IBM) and the ideal ratio mask (IRM) to further enhance the effectiveness of speech separation for single-channel speech separation. While the IBM is advantageous for improving speech intelligibility, it loses more energy from the target signal, and the IRM, while beneficial for enhancing speech quality, retains more interference signal energy. Our proposed method aims to use the strengths of both while mitigating their weaknesses. Experiments on the speech separation in two-speaker tasks were conducted, and three objective metrics were adopted to evaluate our experimental results: the signal-to-distortion ratio (SDR), signal-to-interference ratio (SIR), and signal-to-artifact ratio improvement (SAR). We discussed the impact of different combinations of thresholds on the speech separation performance and suggested thresholds to achieve overall improvement in results. The experimental results show that by setting appropriate thresholds, our proposed method can utilize the advantages of both the IBM and IRM, leading to further improvements in the accuracy of speech separation.

This study demonstrates that by combining the strengths of different masks, the performance of speech separation can be enhanced. In the future, we will consider the characteristics of specific speakers, as well as the differences in different combinations of speakers. By exploring more effective methods of combining with different masks, we aim to further improve the accuracy of speech separation.