Big-Delay Estimation for Speech Separation in Assisted Living Environments

Abstract

Phase wraparound due to large inter-sensor spacings in multi-channel demixing renders the DUET and AdRess source separation algorithms—known for their low computational complexity and effective speech demixing performance—unsuitable for hearing-assisted living applications, where such configurations are needed. DUET is limited to relative delays of up to 7 samples, given a sampling rate of $F_s=16$ kHz in anechoic scenarios, while the AdRess algorithm is constrained to instantaneous mixing problems. The task of this paper is to improve the performance of DUET-type time–frequency (TF) masks when microphones are placed far apart. A significant challenge in assistive hearing scenarios is phase wraparound caused by large relative delays. We evaluate the performance of a large relative delay estimation method, called the Elevatogram, in the presence of significant phase wraparound. We present extensions of DUET and AdRess, termed Elevato-DUET and Elevato-AdRess, which are effective in scenarios with relative delays of up to 200 samples. The findings demonstrate that Elevato-AdRess not only outperforms Elevato-DUET in terms of objective separation quality metrics—BSS_Eval and PEASS—but also achieves higher intelligibility scores, as measured by the Perceptual Evaluation of Speech Quality (PESQ) Mean Opinion Score (MOS) scores. These findings suggest that the phase wraparound limitations of DUET and AdRess algorithms in assistive hearing scenarios involving large inter-microphone spacing can be addressed by introducing the Elevatogram-based Elevato-DUET and Elevato-AdRess algorithms. These algorithms improve separation quality and intelligibility, with Elevato-AdRess demonstrating the best overall performance.

1. Introduction

Selective Hearing (SH) is described in [1] as the ability of listeners to focus on a specific sound source within a competing speaker scenario in their auditory environment. The machine emulation of this process via SH technology involves source localization, separation, and enhancement techniques, as well as unifying frameworks that integrate these methods. Machine emulation of SH seeks to enhance the intelligibility and quality of a target speaker’s voice at the ear of a listening-impaired person. Speech enhancement focuses on improving the perceptual quality of speech that has been degraded by additive noise [2]. These algorithms work to reduce or suppress background noise and are often referred to as noise suppression algorithms [3]. In conjunction with interference suppression (IS), these types of problems are closely related to the problem of source separation (SS) [4]. For better interference reduction performance, researchers often introduce the assumption that each audio signal in a multi-audio scenario is in proximity to at least one microphone channel, that predominantly captures the target signal [5,6,7,8,9]. These methods have extensive applications in assisted living (AL) scenarios where hearing aids (HA) are used [9,10,11], in voice-controlled smart homes [12,13,14] for elderly people and in other speech AL technologies [15,16,17]. Audio SS involves extracting individual sources from a combined audio mixture. This technique has applications in speech enhancement [18], speech recognition [19], and HA devices [20,21].

This paper is motivated by applications in multi-microphone-based auditory localization systems. In the scenario illustrated in Figure 1a, each mixture captures a different version of the speaker, characterized by variations in spatial location, relative attenuation, and delay parameters. These mixtures are processed by powerful speech demixing algorithms capable of isolating a specific voice of interest [22]. The demixed speech is then transmitted in real-time to a wireless HA using Bluetooth, infrared, or other wireless transmission systems [23]. In these systems, the strategic placement of microphones has proven highly effective in addressing challenges such as noise, interference, and reverberation, particularly in the framework of microphone array processing.

Advances in Voice-over-Wireless Sensor Network (VoWSN) systems [24,25,26,27] and Acoustic Sensor Networks [28,29,30,31] are making it increasingly feasible to enable audio communication through microphones and other AL devices [22,32,33], which are wirelessly connected and strategically placed throughout large rooms. The performance of audio processing tasks—including speech enhancement, separation, diarization, and recognition—benefits greatly from the spatial distribution of microphones across a large area in a room [34].

Assistive hearing devices encompass a wide range of listening technologies designed to enhance access to speech signals or to amplify speech of interest for individuals with hearing impairments. These devices address challenges where traditional ear-mounted HAs [35] may be insufficient. Examples include environments with significant background noise, large speaker-to-listener distances, interference, or reverberation. By wirelessly linking to a user’s HA, these devices ensure the delivery of clear, high-quality demixed target speech signals, ensuring a clear and high-quality broadcast of the demixed target speech signal [32]. The prevalence of wireless devices with microphones, including smartphones, laptops, and hands-free kits, is a common feature of modern life. These devices can also be integrated into the network to improve noise reduction in HAs [36]. State-of-the-art solutions often rely on telecoil-based amplification systems. These systems employ wireless remote microphones (RM) paired with HAs [37], using telecoil technology to amplify and transmit the speaker of interest’s voice via Bluetooth, infrared, light waves, or other proprietary wireless technologies [23]. RMs [38] allow clearer reception of the distant speaker of interest at the receiving end which is otherwise achievable when relying on the sound-field near the listener’s ear [39], in other words when the talker speaks close to the listener’s ear. Recent developments aim to help HA users better understand speakers located at a distance [10,11,37,40,41]. Listening can be challenging when there is excessive background noise, interference, reverberation, or a significant distance between the sound source and the person with hearing loss, as these factors can reduce the perceived signal quality [37]. The use of RM has been shown to improve intelligibility in adverse conditions reliably [37,41]. The wireless RM provides a relatively clean reference for speech recognition systems [42]. Additional spatial cues are often available in the form of the relative delay/interaural phase difference (IPD) and the received signal strength/interaural intensity difference (IID). These cues correspond to the panning coefficient in the pan-pot mixing model and are captured on the far-end microphone relative to the RM [38], as shown in Figure 1. Ideally, an RM in conjunction with other distant microphones in a wireless acoustic sensor network (WASN) setting mimics the sound-field energy present at the speaker’s mouth and can transmit it to the listener’s ear in a manner resembling free-field anechoic transmission [39]. Speech enhancement in these settings refers to a class of algorithms that improve the quality and intelligibility of speech transmitted from a far-end microphone—where speech is captured as part of a mixture—to the listeners’ HA, which ultimately broadcasts the demixed target speech of interest.

Spatial cues help us to distinguish sounds arriving at our ears from different talkers and directions separated in space [10]. Understanding how humans interpret these spatial cues can inform the design of microphone array setups that effectively separate sound sources. For instance, a close-talk or near-field microphone array setup, designed to capture sound sources in close proximity, helps the detection of target speech with a relatively high signal-to-noise ratio (SNR). It decreases in effectiveness with increased speaker-to-microphone distance [43,44]. A significant gap, larger than half the smallest wavelength content, ${\lambda }_{\min }$, of the propagating wave, between the RM and the far-end microphone will render the estimation of relative delay cues/IPDs ambiguous [45,46]. This is known as the phase wraparound effect. Phase wraparound occurs because the higher-frequency components of the signal cannot diffract around the microphones positioned along their propagation path [47]. These high frequencies are corrupted and pose a challenge when using the time–frequency (TF) representation [48]. Typical indoor temperatures are often maintained between 20 °C and 25 °C (68 °F and 77 °C) in AL environments. At standard atmospheric pressure, and at a typical room temperature of 20 °C, the speed of sound is 343 m/s. To accurately capture and represent these sound waves digitally, a sufficient sampling rate is required. Given a sampling rate of $F_s=16$ kHz, the maximum frequency content of that signal is $f_{\max }=8$ kHz. The bound on the inter-microphone gap so that phase wraparound does not occur, is commonly known as the spatial Nyquist criterion

$$\begin{array}{c}\hfill d_s<{\displaystyle \frac{{\lambda }_{\min }}{2}}={\displaystyle \frac{c}{2f_{\max }}}.\end{array}$$

(1)

Substituting these values into Equation (1), the maximum permissible separation between sensors is $2.14$ cm. Under similar conditions, setting the inter-microphone gap to be greater than 1 m, $d_s>1$ m, allows the maximum frequency that will be uncorrupted to equal up to $f_{\mathrm{aliasing}}=171.5$ Hz. Frequencies above this maximum frequency, $f_{\mathrm{aliasing}}$, may experience phase wraparound [49]. This corresponds to $97\%$ of the total frequencies being corrupted, resulting in ambiguous delay estimates. Underdetermined speech demixing occurs when the number of sources exceeds the number of microphones. With an increased inter-microphone gap between the RM and a far-end microphone, depicted in Figure 1b, phase wraparound becomes a challenge in near-field settings, leading to two issues: scaling and permutation ambiguities, which complicate underdetermined speech demixing problems. As a result, in the TF domain, a source may exhibit erroneous phases and gains across different TF-bins [50], especially when the frequencies are above $f_{\mathrm{aliasing}}$. This complicates relative delay estimation and consequently source demixing, as clustering-based techniques struggle to assign the TF bins to their correct sources. The authors of [46,49,51] demonstrated that if the IPDs of the low-frequency components that are less than $f_{\mathrm{aliasing}}$, are estimated accurately, they can be used to align the higher frequency TF bins, where phase wraparound occurs to the correct sources. We address the problem of achieving speech separation in competing-speaker assisted living environments, which are characterized by large delays—and consequently phase wraparound—due to the significant physical separation of microphones.

In this paper, we propose two methods of adapting the Degenerate Unmixing Estimation Technique (DUET)-based mask demixing function to operate effectively in anechoic scenarios with large relative delays, of up to 200 samples. These proposed methods are called Elevato-DUET and Elevato-AdRess. Numerical evaluations on real-speech mixtures demonstrate that Elevato-AdRess outperforms Elevato-DUET in most speech separation quality metrics. Furthermore, the techniques introduced in this paper are not affected by the permutation or scaling problems commonly encountered in scenarios involving large relative delays.

This paper is organized as follows. In Section 2, we introduce the mixing model. Section 3 describes the Elevatogram, originally introduced in [5]. Section 4 outlines the contributions of this paper—namely, the Elevato-AdRess and Elevato-DUET algorithms. Finally, Section 5 evaluates the performance of these algorithms in the task of separating the target source, $s_j\left[n\right]$, from a mixture of sources $J=4$.

2. Mixing Model

A reverberant stereo noise-free pair of mixtures is expressed as

$$\begin{array}{cc}\hfill x_1\left[n\right]& =\sum _{j=1}^J{h}_{1j}\left[n\right]\ast s_j\left[n\right],\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill x_2\left[n\right]& =\sum _{j=1}^J{h}_{2j}\left[n\right]\ast s_j\left[n\right],\hfill \end{array}$$

(3)

where ∗ is the linear convolution operator and the filters, $h_{ij}\left[n\right]$, are the channel Received Impulse Responses (RIR) for source $s_j\left[n\right]$ at microphone, $x_i\left[n\right]$. The source-to-microphone distance is $d_{ij}$ and K is the number of DFT points. For an anechoic or low reverberant scenario, the mixing filter for the $i^{th}$ channel, $h_{ij}\left[n\right]$, simplifies to a combination of delays, ${\delta }_{ij}$, and attenuations, ${\alpha }_{ij}$, which are expressed as,

$${\alpha }_{ij}={\displaystyle \frac{1}{\sqrt{4\pi }{d}_{ij}}},$$

(4)

where $d_{ij}$ is the distance between the $j^{th}$ source and the $i^{th}$ microphone [49]. The Acoustic Transfer Function (ATF) of the $j^{th}$ source, which is the frequency domain transform of the RIR, which is called its steering vector in a scenario with I microphones is,

$$\begin{array}{c}\hfill {\mathbf{d}}_j\left[k\right]=\left[{\displaystyle \frac{1}{\sqrt{4\pi }{d}_{1j}}}{e}^{-j{\displaystyle \frac{2\pi }{K}}k{\displaystyle \frac{d_{1j}}{c}}{F}_s},{\displaystyle \frac{1}{\sqrt{4\pi }{d}_{2j}}}{e}^{-j{\displaystyle \frac{2\pi }{K}}k{\displaystyle \frac{d_{2j}}{c}}{F}_s},\dots ,{\displaystyle \frac{1}{\sqrt{4\pi }{d}_{Ij}}}{e}^{-j{\displaystyle \frac{2\pi }{K}}k{\displaystyle \frac{d_{Ij}}{c}}{F}_s}\right].\end{array}$$

(5)

The gain normalized ATF of the $j^{th}$ source, called the Relative Transfer Function (RTF) is,

$$\begin{array}{c}\hfill {\mathbf{d}}_j^{\mathrm{Rel}}\left[k\right]=\left[1,{\displaystyle \frac{d_{1j}}{d_{2j}}}{e}^{-j{\displaystyle \frac{2\pi }{K}}k{\displaystyle \frac{(d_{2j}-d_{1j})}{c}}{F}_s},\dots ,{\displaystyle \frac{d_{1j}}{d_{Ij}}}{e}^{-j{\displaystyle \frac{2\pi }{K}}k{\displaystyle \frac{(d_{Ij}-d_{1j})}{c}}{F}_s}\right],\end{array}$$

(6)

and can be uniquely determined by the relative gain and delay estimates [52], in a low reverberant room [53]. Equation (6) helps us fix the scaling problem [54]. Substituting Equation (4) in Equation (6) gives the RTF,

$$\begin{array}{cc}\hfill {\mathbf{d}}_j^{\mathrm{Rel}}\left[k\right]=& \left[1,{\alpha }_{2j}{e}^{-j{\displaystyle \frac{2\pi }{K}}k{\delta }_{2j}},\dots ,{\alpha }_{Ij}{e}^{-j2{\displaystyle \frac{2\pi }{K}}k{\delta }_{Ij}}\right]\hfill \end{array}$$

(7)

where ${\delta }_{ij}={\displaystyle \frac{\left(d_{ij}-d_{1j}\right)}{c}}{F}_s,$ is the relative delay measured in samples and the attenuation of the $j^{th}$ source impinging on the $i^{th}$ microphone is ${\alpha }_{ij}={\displaystyle \frac{d_{1j}}{d_{Ij}}}$, which is bounded by $0\le {\alpha }_{ij}\le 1$, and is used to define an anechoic mixing model. The anechoic mixing model is a special case of the reverberant model. In a room which has low reverberance, or which has no echoes, the RTF in Equation (7) can be uniquely determined by the relative gain and delay estimates [52,53]. The terms relative gain, attenuation and IID are used interchangeably in the literature.

Conventional relative delay estimation techniques, for example, the Generalized Cross-Correlation-PHAse Transform (GCC)-PHAT [55], Steered Power Response (SRP)-PHAT [56], and MUltiple SIgnal Classification (MUSIC) [57] are designed to estimate the loudest source in an indoor acoustic environment. The low power of the target speech source often impedes the accuracy of the delay estimate [58] in a multi-talker scenario because the phase information is lost due to other speech interference [59]. Joint multi-source delay estimation is challenging when two sources are spatially close to one another [46,60,61]. In addition, speech enhancement techniques are not useful when the interferer is in proximity to the target speaker of interest [62]. Recent work in [4] demonstrated that the accuracy of widely used delay estimation techniques decreased when tasked with estimating the relative delays of all constituent sources simultaneously. The advantage of a close-talk setup is the ability to estimate the most prominent peak associated with a relative delay [63], which corresponds to the reference microphone positioned near the target speaker’s mouth. This provides a reliable estimate of the IPD for the speaker of interest, which can then be used to parameterize a TF mask to isolate the speech effectively.

Assuming that an arbitrary source, $s_j\left[n\right]$, is in physical proximity to any one RM, as shown in Figure 1b, then the corresponding discrete-time, anechoic stereo-mixing scenario, which consists of J sources is,

$$\begin{array}{cc}\hfill x_1\left[n\right]& =s_j\left[n\right],\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill x_2\left[n\right]& =\sum _{j=1}^J{\alpha }_{2j}{s}_j[n-{\delta }_{2j}],\hfill \end{array}$$

(9)

where the discrete time index is n, where $1\le n\le N$ and $n\in {\mathbb{Z}}_{+}$. Given a binaural mixture, the relative attenuation and delay coefficients are redefined, ${\alpha }_{2j}$ as ${\alpha }_j$ and ${\delta }_{2j}$ as ${\delta }_j$, for simplicity. In the mixture $x_2\left[n\right]$, the source, $s_j\left[n\right]$, is attenuated and delayed by ${\alpha }_j$ and ${\delta }_j$, respectively, relative to $x_1\left[n\right]$. The relative attenuation lies in the range $0\le {\alpha }_j\le 1$, and the relative delay ${\delta }_j$ is measured in samples. The TF transform of $x_1\left[n\right]$ provides the mapping as follows, ${\mathbf{X}}_1:x_1\left[n\right]\in \mathbb{R}\to {\mathbf{X}}_1[k,\tau ]\in \mathbb{C}$, where the discrete frequency is k, and where $1\le k\le K$. The discrete-time frame is $\tau$, where $1\le \tau \le T$. Similarly, $x_2\left[n\right]$ is mapped to ${\mathbf{X}}_2[k,\tau ]$, using the TF transform stated in [48].

It is assumed that the target speaker is always the most intense among all speakers present. As proposed in [4], speech enhancement with interference reduction using an RM can enable speech demixing. The demixing of a mixture of voice signals at the far-end microphone is illustrated in Figure 1b. In a simple anechoic model, an acoustic speech wave at one microphone differs from the reference RM signal in terms of IPD and IID cues. We hypothesize that DUET-based masking techniques, which include the classical DUET [64] and AdRess [65] algorithms—where the latter was originally designed for separating music mixed using a pan-pot model—can be adapted to a large-delay anechoic model, with delays of up to 200 samples. The AdRess technique is a brute force version of the DUET mask, where the IID cues are localized on a frequency–azimuth plane relative to a gain scale.

In the time domain, the silent interval between two interacting speakers is typically small. This causes the majority of the signal to have coefficients which are non-zero [66]. In the TF domain, a small percentage of the TF coefficients contain a majority of the energy of the acoustic signal [67]. Speech is typically sparse and disjoint in the TF domain [68]. The TF representation is commonly employed for speech processing because sources tend to exhibit less overlap in the TF domain compared to the temporal waveform [69,70]. While speech in the time domain is not sparse during active speech, it is typically sparse in the TF domain. This implies that a small percentage of the TF Gabor coefficients capture a large part of the energy. Specifically, the magnitudes of most of these Gabor coefficients are generally small. According to the authors of [69,71] it is unlikely that multiple speech signals will overlap in a particular TF bin of the mixture of speech utterances. This characteristic makes the signals approximately Windowed-Disjoint Orthogonal (WDO) [69,71]. The authors in [67] observed that the TF bins that contain $90\%$ of the energy of the $j^{th}$ source contain less than $1\%$ of the energy of all the interference combined together.

Strict, pairwise WDO between two speech utterances, ${\mathbf{S}}_1[k,\tau ]$ and ${\mathbf{S}}_2[k,\tau ]$, separated in space, in a channel mixture is expressed as,

$$\begin{array}{c}\hfill {\mathbf{S}}_1[k,\tau ]{\mathbf{S}}_2[k,\tau ]=0,\forall [k,\tau ],\mathrm{and}{\mathbf{S}}_1\ne {\mathbf{S}}_2.\end{array}$$

(10)

In real-mixtures, these utterances are sparse. They do not satisfy the condition in Equation (10), but they are approximately-sparse. A simple explanation is obtained by calculating the Log-Max approximation in the TF domain as,

$$\begin{array}{cc}\hfill {\mathbf{X}}_i[k,\tau ]& ={\mathbf{S}}_1[k,\tau ]+{\mathbf{S}}_2[k,\tau ]\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & ={\mathbf{S}}_1[k,\tau ]\left(1+{\displaystyle \frac{{\mathbf{S}}_2[k,\tau ]}{{\mathbf{S}}_1[k,\tau ]}}\right).\hfill \end{array}$$

(12)

Taking the natural-logarithm on both sides produces

$$\begin{array}{cc}\hfill ln\left({\mathbf{X}}_i[k,\tau ]\right)& =ln\left({\mathbf{S}}_1[k,\tau ]\right)+ln\left(1+{\displaystyle \frac{{\mathbf{S}}_2[k,\tau ]}{{\mathbf{S}}_1[k,\tau ]}}\right).\hfill \end{array}$$

(13)

If ${\mathbf{S}}_1[k,\tau ]>>{\mathbf{S}}_2[k,\tau ]$, then ${\displaystyle \frac{{\mathbf{S}}_2[k,\tau ]}{{\mathbf{S}}_1[k,\tau ]}}<<1$ and Equation (13) reduces to

$$\begin{array}{c}\hfill ln\left({\mathbf{X}}_i[k,\tau ]\right)\approx ln\left({\mathbf{S}}_1[k,\tau ]\right),\because ln\left(1+{\displaystyle \frac{{\mathbf{S}}_2[k,\tau ]}{{\mathbf{S}}_1[k,\tau ]}}\right)\to 0.\end{array}$$

(14)

Similarly, if ${\mathbf{S}}_2[k,\tau ]>>{\mathbf{S}}_1[k,\tau ]$, then taking ${\mathbf{S}}_2[k,\tau ]$ as a factor in Equation (11), it can be shown that

$$\begin{array}{c}\hfill ln\left({\mathbf{X}}_i\left[n\right]\right)\approx ln\left({\mathbf{S}}_2\left[n\right]\right),\because ln\left(1+{\displaystyle \frac{{\mathbf{S}}_1[k,\tau ]}{{\mathbf{S}}_2[k,\tau ]}}\right)\to 0.\end{array}$$

(15)

Consequently, the expression

$$ln\left({\mathbf{X}}_i[k,\tau ]\right)\approx \{\begin{array}{cc}\hfill ln\left({\mathbf{S}}_2\left[n\right]\right),& \mathrm{if}{\mathbf{S}}_2[k,\tau ]>>{\mathbf{S}}_1[k,\tau ]\\ \hfill ln\left({\mathbf{S}}_1\left[n\right]\right),& \mathrm{if}{\mathbf{S}}_1[k,\tau ]>>{\mathbf{S}}_2[k,\tau ]\end{array}$$

assumes that the source with the maximum energy among the constituent sources is essentially the sole occupant of a TF bin, leading to the definition of approximate WDO,

$$ln({\mathbf{S}}_1[k,\tau ]+{\mathbf{S}}_2[k,\tau ])=\max \left(ln\left({\mathbf{S}}_1[k,\tau ]\right),ln\left({\mathbf{S}}_2[k,\tau ]\right)\right).$$

(16)

The approximate WDO description in Equation (16), accompanied by the idea of the dominance of a particular speech utterance in a TF bin, is a good approximation of the strict WDO criterion given in Equation (10), when the sources are i.i.d. Gaussian. Similarly, when the number of sources present in Equation (11) is N, it can be stated that the sources are N-way W-disjoint orthogonal and predominant in Equation (16), $\forall k,\tau ,j$.

3. The Elevatogram for Large Delay Estimation

The Elevatogram was introduced in [5]. In previous contributions, it was demonstrated that in an anechoic condition without any environmental noise, the Elevatogram can estimate relative delays up to more than 500 samples, given a sampling frequency of, $F_s=16$ kHz [5,8]. The TF representation of the mixtures, $x_1\left[n\right]$ and $x_2\left[n\right]$, in Equations (8) and (9), are given as ${\mathbf{X}}_1[k,\tau ]$ and ${\mathbf{X}}_2[k,\tau ]$, respectively. The Cross-Power Spectrum (CPS) is calculated as

$$\begin{array}{c}\hfill \mathbf{X}[k,\tau ]={\mathbf{X}}_1[k,\tau ]{\overline{\mathbf{X}}}_2[k,\tau ],\end{array}$$

(17)

where the complex conjugate of ${\mathbf{X}}_2[k,\tau ]$ is denoted as ${\overline{\mathbf{X}}}_2[k,\tau ]$. The normalized CPS contains the phase information for a given delay, ${\delta }_j$, and is expressed as

$$\begin{array}{c}\hfill \Delta \varphi [k,\tau ]=\angle \mathbf{X}[k,\tau ]=\angle {\mathbf{X}}_1[k,\tau ]-\angle {\mathbf{X}}_2[k,\tau ]+\Omega {\delta }_j+2\pi r,\end{array}$$

(18)

where $r\in \mathbb{Z}$ comes from the modulo-$2\pi$ phase ambiguity [72], and $\Delta \varphi [k,\tau ]\in \mathbb{R}$ and $\Omega =2\pi k/K$. The parameter, $\Delta \varphi [k,\tau ]$, can be modeled as a wrapped scaled-Gaussian Mixture Model (scaled-GMM) or Von-Mises noise distribution [73,74], because the phase is a circular variable that is bounded in the interval $(-\pi ,\pi ]$. The random variable $\Delta \varphi [k,\tau ]$ is zero-mean. It has a periodicity of $2\pi$ and is unimodal in one cycle [75] and i.i.d. A histogram of the $k^{th}$ discrete frequency row of $\Delta \varphi [k,:]$ was generated in the Elevatogram in [5]. The bins of the histogram are given by the vector ${[-\pi ,\dots ,\pi ]}^T$, which has L elements, each corresponding to a quantization level, which are uniformly separated, and each bin is spaced by ${\Delta }_d$. If the bins are indexed by $\tilde{\varphi }$, then the set of TF bins contributing to any $\widehat{\varphi }$, is obtained using

$$I\left[\widehat{\varphi }\right]:=\{\{k,\tau \}:|\Delta \varphi [k,\tau ]-\tilde{\varphi }|<{\Delta }_d\}.$$

(19)

The higher the cardinality of $\left|I\right[\tilde{\varphi }\left]\right|$, the more intensely that bin $\widehat{\varphi }$ has been activated in the histogram, ${\mathbf{h}}_k^T$. The histogram is generated as follows,

$${\mathbf{h}}_k=\mathrm{hist}\left(\Delta \varphi [k,:],L\right)\in {\mathbb{R}}^{1\times L}.$$

(20)

The Elevatogram [5] computes a histogram for each of the K frequencies. It generates a matrix, $\mathbf{P}\in {\mathbb{R}}^{L\times K}$, which was also called the Elevatogram by the authors of [5], and is obtained by concatenation resulting in a phase–frequency matrix,

$$\begin{array}{c}\hfill \mathbf{P}=[{\mathbf{h}}_1^T,{\mathbf{h}}_2^T,{\mathbf{h}}_3^T,\dots {\mathbf{h}}_K^T]\in {\mathbb{R}}^{L\times K}.\end{array}$$

(21)

The slanted lines in the example shown in Figure 2a, which depict the distribution of phase with respect to frequency, represent a single line that undergoes phase wraparound at $1,3,5$ and 7 kHz. The larger the delay, the more often this phase wraparound occurs. The location of the set of most significant collinear points forming a straight line in the phase–frequency domain was determined in [5], as shown in Figure 2b. This concept is analogous to the distance–angle accumulator in the Hough transform [76], which receives the highest activation. This value is marked in green and denoted as ${\varphi }_{\max }$. The relative delay produced by the Elevatogram [5] is

$${\widehat{\delta }}_j=-{\displaystyle \frac{K}{L}}tan\left({\varphi }_{\max }\right).$$

(22)

In general, the phase wraparound issue prevents the IPD cues from being mapped to a unique delay, ${\delta }_j$. Using the closed-form solution in Equation (22), the relative delay can be estimated unambiguously. This explains the success of the Elevatogram approach [5]. For better estimation, the base histogram matrix, $\mathbf{P}$, was tiled by the author of [5] by concatenating tiles in a $\mathcal{T}\times \mathcal{T}$ grid. Each tile was a replica of $\mathbf{P}$ and concatenating them adjacent to and below the base $\mathbf{P}$ gave the consolidated tiled histogram [5]. This procedure can be interpreted as a Maximum Likelihood Estimator (MLE). Given a set of independent and identically distributed (i.i.d.) N pixel input features: $X=${$x_1,x_2,\dots ,x_N$}, signifying N significant collinear points in the phase-frequency feature space which are mapped to an accumulator in the parametric [$\rho ,\varphi$]-space shown in Figure 2a, the joint distribution is defined in Appendix A and defined here

$$L\left([\rho ,\varphi ]\right)=\prod _{i=1}^{N}p\left(x_i|[\rho ,\varphi ]\right),x_i\in X.$$

(23)

The log-likelihood of which is expressed as,

$$\mathcal{L}[\rho ,\varphi ]=logL\left([\rho ,\varphi ]\right)=\sum _{i=1}^{N}logp\left(x_i|[\rho ,\varphi ]\right).$$

(24)

As more features (pixels) are taken into account, the PDF in the Hough space, given N features, will change from a uniform distribution, where all accumulators are equally likely, to a maximum at a particular accumulator, $[\widehat{\rho },\widehat{\varphi }]$ in Figure 2b as the MLE, which is computed as

$$\begin{array}{c}\hfill [\widehat{\rho },\widehat{\varphi }]=\underset{\rho ,\varphi }{argmax}\mathcal{L}[\rho ,\varphi ].\end{array}$$

(25)

4. TF Masking

Widely recognized components of source separation (SS) approaches include Estimation of Signal Parameters via Rotational Invariant Techniques (ESPRIT) [77], DUET [67], TIme Frequency Ratio Of Mixture (TIFROM) [78] and the Direction Estimation of Mixing matrIX (DEMIX) [79] algorithms. DUET-ESPRIT (DESPRIT) [80] is an extension of ESPRIT. A family of power-weighted estimators was introduced in [71], exploiting the sparsity of sources in a specific transform domain [81], in this case the TF domain. The core principle of Independent Component Analysis (ICA)-based methods is to leverage the sparsity of sources in a specific transform domain [82]. The inherent sparsity of the sources implies that they are already separated, enabling SS to be performed using a binary or hard mask. The essence of ICA-based methods lies in the exploitation of the sparsity of the sources in the transform domain [82]. If the sources are sparse, this implies that they are already separated [68]. Speech demixing can then be achieved using a binary or hard mask, as shown in Equation (26). The mask is a weighting matrix that corresponds to a filter for the target source from the mixture [20]. Hard-masking techniques assume that a TF-bin belongs to one source. According to the authors in [67], if all the TF bins that correspond to a particular source are determined, then the source is already separated.

$$\mathbf{M}[k,\tau ]=\left\{\begin{array}{c}1,\mathrm{i}\mathrm{f}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{T}\mathrm{F}-\mathrm{b}\mathrm{i}\mathrm{n}\in {\mathbf{S}}_j\hfill \\ 0,\mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}.\hfill \end{array}\right.$$

(26)

The mask, $\mathbf{M}[k,\tau ]$, has TF bins set to 1 for all components belonging to the speaker of interest, ${\mathbf{S}}_j$. All other TF bins in $\mathbf{M}[k,\tau ]$ belong to the background interference and are set to 0. The hard-mask AdRess algorithm was extended to handle anechoic scenarios in [83]. A soft-masking variant of AdRess, is known as Reformulated-AdRess (Redress) [84] because it tackles the hard-masking short-coming of AdRess. It was motivated by prior work on single channel feature learning in the presence of background noise [85]. It was adapted for anechoic conditions in [86]. These techniques are computationally efficient soft-masking techniques for SS. Unlike hard-masking methods, which use binary decisions, soft-masking approaches generate a ratio mask $\mathbf{M}[k,\tau ]$ with values ranging between $0\le \mathbf{M}[k,\tau ]\le 1$. This approach distributes the energy contributions in $\mathbf{M}[k,\tau ]$ across sources, based on the likelihood of their presence in the mixture [87]. TF-masking has seen application in Automatic Speech Recognition (ASR) systems because these systems involve a step to isolate the target from its acoustic background.

4.1. Elevato-AdRess: Separation via Frequency–Azimuth Plane

The classical AdRess algorithm [65] is often used for music demixing, particularly when the sources are mixed using a pan-pot approach. It isolates sources by localizing the IID cues on a frequency–azimuth plane. This method separates music from the TF mixture by employing a form of spectral energy cancellation technique, which uses attenuations as a spatial cue. They are obtained from the magnitude spectra, $|{\mathbf{X}}_i[k,\tau ]|$, where ${\mathbf{X}}_i$ is the $i^{th}$ channel mixture. The AdRess algorithm [65] was licensed for integration into Sony’s SingStar game on the PlayStation 3. Subsequently, it was licensed to Riffstation, which was later acquired by Fender and utilized by millions of users between 2012 and 2018. Each source is given a position in the stereo field using an azimuth location. In the sound engineering and music production industries these panning coefficients are used to create multichannel mixtures [65,88]. These coefficients scale the contribution of each source in each of the available channels.

We extended this approach to an anechoic speech mixing model, introducing the D-AdRess technique in [83], which considers the mixing model in Equations (8) and (9). D-AdRess uses the magnitude spectra, $|{\mathbf{X}}_i[k,\tau ]|$ and the phase spectra, $\angle {\mathbf{X}}_i[k,\tau ]$. D-AdRess was restricted to scenarios with a relative delay of up to 5 samples. The primary contribution of the present paper is an extension of the AdRess technique. It extends AdRess to large delay scenarios where the delays are up to 200 samples. The resulting algorithm is called Elevato-AdRess, acknowledging its reliance on the Elevatogram for accurate estimation of large delays [5]. Elevato-AdRess is summarized in Algorithm 1. The AdRess technique has garnered widespread adoption in recent research [89,90,91], and has been improved in subsequent contributions [9,84]. In addition, it has also been cited in a patent for an acoustic noise monitoring framework [92]. The proposed method, Elevato-AdRess, operates as a beamforming technique that maximizes the power of the source arriving from a specific direction, corresponding to ${\widehat{\delta }}_j$, using a pair of microphones. The Signal-to-Noise Ratio (SNR) is the ratio of the target source power to the combined power of all interfering speech sources. This approach is equivalent to maximizing the SNR, as described in [61]. The use of relative delay estimates for beamforming was reported in [93]. Prior information regarding the direction of the target and the interference helps implement robust IS techniques for speech enhancement [44].

To introduce Elevato-AdRess, we consider the two-synthetic-signal example that served as motivation in [84]. The first signal, $s_1\left[n\right]$, is composed of sinusoids with frequencies, $k_1=100$ Hz and $k_3=300$ Hz, and the second signal, $s_2\left[n\right]$, has components of frequency, $k_2=200$ Hz and $k_3=400$ Hz. The resulting sources

$$\begin{array}{cc}\hfill s_1\left[n\right]& =sin\left(2\pi k_1{\displaystyle \frac{n}{F_s}}\right)+sin\left(2\pi k_3{\displaystyle \frac{n}{F_s}}\right),\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill s_2\left[n\right]& =sin\left(2\pi k_2{\displaystyle \frac{n}{F_s}}\right)+sin\left(2\pi k_4{\displaystyle \frac{n}{F_s}}\right),\hfill \end{array}$$

(28)

are mixed via Equations (8) and (9) with the attenuation coefficients ${\alpha }_1=0.8$ and ${\alpha }_2=0.2$. The authors of [84] localized the IID cues on a frequency–gain plane, $\mathbf{A}=[{\mathbf{A}}_1,{\mathbf{A}}_2]\in {\mathbb{R}}^{K\times M}$, as illustrated in Figure 3a, by scaling $|{\mathbf{X}}_1|$ relative to $|{\mathbf{X}}_2|$ and vice versa, with respect to an attenuation range g, where $0\le g\le 1$. This range is divided into M equally spaced levels. The cancellation approach described in [5] results in the difference between the magnitudes of the Short-Time Fourier Transform (STFT) representations of the constituent sources to be zero [48]. A plane is constructed by finding the best $\widehat{g}$ for a particular source, where

$$\begin{array}{cc}\hfill \widehat{g}\leftarrow \mathrm{f}\mathrm{i}\mathrm{n}\mathrm{d}& \left({\mathbf{A}}_1=|{\mathbf{X}}_1[k,\tau ]-g{\mathbf{X}}_2[k,\tau ]|\approx 0,\right.\hfill \\ \hfill & \left.\mathrm{a}\mathrm{n}\mathrm{d}{\mathbf{A}}_2=|{\mathbf{X}}_2[k,\tau ]-g{\mathbf{X}}_1[k,\tau ]|\approx 0\right),\hfill \end{array}$$

(29)

where the spectral energies of the sources are canceled, according to [84], producing

$${\mathbf{A}}_1=\left\{\begin{array}{c}{\alpha }_1={\displaystyle \frac{1}{g}}\mathrm{if}k=k_1\hfill \\ {\alpha }_2={\displaystyle \frac{1}{g}}\mathrm{if}k=k_3\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.$$

(30)

$${\mathbf{A}}_2=\left\{\begin{array}{c}{\alpha }_1=g,\mathrm{if}k=k_2\hfill \\ {\alpha }_2=g,\mathrm{if}k=k_4\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.$$

(31)

provided $g<0$. In Figure 3a, it is observed that a null forms at $\widehat{g}={\alpha }_1=0.2$ for the frequencies of $s_1\left[n\right]$, $k=100$ Hz and $k=300$ Hz. Similarly, the source $s_2\left[n\right]$ contains the nulls at $\widehat{g}={\alpha }_2=0.8$ for the frequencies, $k=200$ Hz and $k=400$ Hz, for non-overlapping frequencies. This frequency–gain plane is also called the frequency–azimuth plane because it provides information on the source’s azimuth in the stereo mixture. The foundational step is to isolate the target source. A two-microphone scenario gives the azimuth location of the source and not the elevation. Peaks constructed at these nulls determine the energy content of the sources. If they overlap, the corresponding null forms at $\widehat{g}\approx {\displaystyle \frac{{\alpha }_1+{\alpha }_2}{2}}$ or ${\displaystyle \frac{2}{{\alpha }_1+{\alpha }_2}}$, subject to $|\widehat{g}|\le 1$. The initial, detailed discussion justifying this assertion is given in [94]. Figure 3c shows that the sources overlap at 300 Hz and have peaks at $\widehat{g}\approx 0.5$. AdRess is restricted to pan-pot mixing scenarios. Detection of peaks at $\widehat{g}$ becomes challenging in anechoic mixing scenarios. We formulate a technique for delay cancellation in the second mixture, ${\mathbf{X}}_2$, using the delay estimated via the Elevatogram [5] in Equation (22) in [83], and scale ${\mathbf{X}}_2$, based on the approach in [94] according to

$${\mathbf{X}}_2=e^{+j\omega {\widehat{\delta }}_j}\odot {\mathbf{X}}_2.$$

(32)

This operation localizes the spectral energies at accurate gain locations on the frequency–azimuth plane. Extracting the subportion of $\mathbf{A}[:,:]$ where these peaks occur corresponding to a particular source and repeating this for all the available time frames, T, results in an augmented TF representation. Inverting this TF representation gives the demixed source in the temporal domain. The spectral energy of sources at overlapping frequency locations cannot be accurately extracted using this approach. For real speech signals, the spectral energies at these overlapping frequencies are dispersed across $\mathbf{A}$ in Figure 3, making their precise estimation ambiguous.

Spatial and spectral information must be highly correlated for AdRess to function effectively in an anechoic scenario. Any inaccuracy in estimating the relative delay can result in discrepancies when localizing the spectral energy of the corresponding source on the frequency–azimuth plane, ultimately hindering the technique’s ability to isolate the target speaker.

Algorithm 1 Elevato-AdRess algorithm

1: Find the delay estimate, ${\widehat{\delta }}_j$, using Equation (22) as per the Elevatogram [5]. 2: Subtract the delayed sources in ${\mathbf{X}}_2$ using Equation (32). 3: Extract the peaks corresponding to the target source using Equation (29) and generate an augmented TF representation. 4: Compute the inverse TF (ITF) transform to recover the target utterance in time domain.

4.2. Elevato-DUET

Assuming that speech sources are sparse in the TF mixture suggests that at any bin, one source is active. The RTF in Equation (6) for a stereo-anechoic setting, where the number of sensors is $I=2$, in Equations (8) and (9), gives us the mixing model,

$$\left[\begin{array}{c}{\mathbf{X}}_1[k,\tau ]\\ {\mathbf{X}}_2[k,\tau ]\end{array}\right]=\left[\begin{array}{c}1\\ {\alpha }_j{e}^{-j{\displaystyle \frac{2\pi }{K}}k{\delta }_j}\end{array}\right]{\mathbf{S}}_j[k,\tau ],$$

(33)

where ${\mathbf{S}}_j[k,\tau ]$ is the speaker of interest that must be isolated from the mixture and directed to the hearing-aid. In many papers where the DUET-based sparsity assumption is made, for example [95,96,97,98,99], the TF-bins where one source is active are referred to as Single Source Points (SSPs). This assumption is restricted to noiseless anechoic scenarios. Given a sampling rate of $F_s=16$ kHz, DUET can accurately estimate delays of up to 7 samples. A recent contribution in [91] extended DUET to echoic scenarios and used a Head Related Transfer Function (HRTF) data lookup table, where the spatial cues of different speakers in the room relative to the listener are predefined. This approach is an extension of the TF dictionary approach in [95] which constructs spatial signatures for different room locations. The Google-held patent [100] underlines the efficacy, quality and robustness of DUET-based masking techniques. The classical DUET technique described in [64] constructs a 2-D weighted histogram from the ratio of the TF mixtures,

$$\begin{array}{c}\hfill {\displaystyle \frac{{\mathbf{X}}_2[k,\tau ]}{{\mathbf{X}}_1[k,\tau ]}}\approx {\alpha }_j{e}^{-j{\displaystyle \frac{2\pi }{K}}k{\delta }_j},\forall [k,\tau ]\mathrm{belongs}\mathrm{to}{\mathbf{S}}_j.\end{array}$$

(34)

The two axes represent a range of relative attenuations and delays, which are denoted by the discrete indices, $\tilde{\alpha }$ and $\tilde{\delta }$. Each axis is evenly divided with resolution widths of ${\Delta }_{\alpha }$ and ${\Delta }_{\delta }$. The set of points contributing to a specific location, $[\alpha ,\delta ]$, in the histogram are defined as

$$I[\alpha ,\delta ]:=\left\{|\alpha [k,\tau ]-\tilde{\alpha }|<{\Delta }_{\alpha },|\delta [k,\tau ]-\tilde{\delta }|<{\Delta }_{\delta }\right\}.$$

(35)

This 2-D histogram,

$$H[\alpha ,\delta ]:=\int I[\alpha ,\delta ]d\alpha d\delta ,$$

(36)

aggregates membership of the set of $(\alpha ,\delta )$ that belong to one source, ${\mathbf{S}}_j\left[n\right]$. The most prominent peaks on $H[\alpha ,\delta ]$ are the estimates of the mixing parameters, $({\widehat{\alpha }}_j,{\widehat{\delta }}_j)$, associated with the J sources, $\{({\widehat{\alpha }}_j,{\widehat{\delta }}_j),j=1,2\dots ,J\}$. The underlying assumption is that the peaks located are sufficiently far apart and identifiable. More details about this technique can be found in [64].

In large delay scenarios, where $d_s>{\lambda }_{\min }/2$, the DUET mixing parameter estimation step using the 2-D histogram plane is often not the best approach. The Elevatogram described in [5] provides good estimates of ${\widehat{\delta }}_j$ via Equation (22). This estimate can be used to subtract the delayed source in the second mixture using Equation (32) via the AdRess step in Equation (29), by estimating the attenuation, ${\widehat{\alpha }}_j$. This relative attenuation–delay pair can be used to generate a mask that partitions the TF-mixture

$$\mathbf{J}[k,\tau ]:=\underset{j}{\mathrm{argmin}}{\displaystyle \frac{{\left|{\widehat{\alpha }}_j{e}^{-j{\widehat{\delta }}_j\Omega }{\mathbf{x}}_1[k,\tau ]-{\mathbf{x}}_2[k,\tau ]\right|}^2}{1+{\widehat{\alpha }}_j^2}}.$$

(37)

The variable $\Omega ={\displaystyle \frac{2\pi }{K}}k$, is the angular frequency. It is in the range $-\pi \le \Omega \le +\pi$. This procedure is iterated over all the available parameter pairs, $({\alpha }_j,\widehat{{\delta }_j})$, to compute the TF mask,

$$\mathbf{M}[k,\tau ]=\left\{\begin{array}{c}1,\mathrm{i}\mathrm{f}\mathbf{J}[k,\tau ]={\mathbf{S}}_j\hfill \\ 0,\mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e},\hfill \end{array}\right.$$

(38)

where $\mathbf{M}[k,\tau ]\in \{1,0\}\in {\mathbb{R}}_{+}$. Sources are demixed by combining the mask and the MLE estimate using

$${\widehat{\mathbf{S}}}_j[k,\tau ]=\mathbf{M}[k,\tau ]\left({\displaystyle \frac{{\mathbf{X}}_1[k,\tau ]+{\widehat{\alpha }}_j{e}^{-j{\widehat{\delta }}_j\omega }{\mathbf{X}}_2[k,\tau ]}{1+{\widehat{\alpha }}_j^2}}\right),$$

(39)

where ${\widehat{\mathbf{S}}}_j[k,\tau ]$ is the recovered target speech utterance. This illustrates that Elevato-DUET can demix the target speaker experiencing a large relative delay.

The quality of any TF-mask, evaluated using the metric Preserved-Signal Ratio (PSR), is computed using

$$\mathrm{PSR}:={\displaystyle \frac{\left|\right|\mathbf{M}[k,\tau ]{\mathbf{S}}_j[k,\tau ]|{|}^2}{\left|\right|{\mathbf{S}}_j[k,\tau ]|{|}^2}},$$

(40)

and is defined as a measure of how well it preserves the energy of, $s_j\left[n\right]$. In addition, the Source-to-Interference Ratio (SIR) is given as

$$\mathrm{SIR}={\displaystyle \frac{{∥\mathbf{M}[k,\tau ]{\mathbf{S}}_j[k,\tau ]∥}^2}{{∥{\sum }_{p=1}^J{\mathbf{S}}_p[k,\tau ]∥}^2}},\mathrm{if}p\ne j$$

(41)

where ${\sum }_{p=1}^J{\mathbf{S}}_p[k,\tau ]$ is the sum of the interference. The WDO is calculated as

$$WDO=\mathrm{PSR}-{\displaystyle \frac{\mathrm{PSR}}{\mathrm{SIR}}}.$$

(42)

WDO has a maximum value of 1 and is in the range $0<\mathrm{WDO}<1$. A WDO value that is less than 0 implies that the mask rejects the target source energy and that the voice is barely intelligible. The PSR and SIR are measured in dB. In Equation (33), observe that the $\mathrm{WDO}\approx 1$ when the $\mathrm{SIR}\approx \infty$ signifying good separation performance. The mask, $\mathbf{M}[k,\tau ]$, preserves the energy of the speech of interest while suppressing the interference. A WDO of approximately 0 means that either the PSR is approximately zero or that the SIR is approximately 1. The former implies the mask rejects the spectral energy of the target speaker, and the latter means ${∥{\mathbf{S}}_j[k,\tau ]∥}^2={∥{\sum }_{p=1}^J{\mathbf{S}}_p[k,\tau ]∥}^2$, that is, the mask results in the source having equal energy as the interference. In practical demixing scenarios, the ideal WDO cannot be achieved in a strict sense. WDO is considered in an approximate sense. The greatest obtainable WDO drops to less than 1 as the number of sources in the mixture increases. Calculating the PSR of Elevato-AdRess is not as straightforward as Elevato-DUET because the mask $\mathbf{M}[k,\tau ]$ is not derived in the form of a binary matrix. Instead, it looks to extract the spectral energies that are localized on the frequency–azimuth plane in Figure 3, and then to create an augmented TF representation. Applying the Inverse TF (ITF) transform gives us the estimated utterance.

5. Results

Anechoic stereo mixtures consisting of up to four sources were generated using randomly selected speech utterances from a total of 25,200 files in the TIMIT corpus [101]. The evaluation was carried out using a TF window with a $K=2048$-sample Hamming window and a $50\%$ overlap. Algorithms 1 and 2 were implemented in MATLAB 2021a.

Algorithm 2 Elevato-DUET algorithm

1: Find the delay estimate, ${\widehat{\delta }}_j$, using Equation (22) as per the Elevatogram [5]. 2: Subtract the delayed sources in ${\mathbf{X}}_2$ using Equation (32). 3: Find the attenuation estimate, ${\widehat{\alpha }}_j$, using Equation (29). 4: Using the estimated (${\widehat{\alpha }}_j$, ${\widehat{\delta }}_j$) pairs corresponding to the sources, compute the TF mask using Equations (26) and (37). 5: Compute the ITF transform.

The goal is to separate $s_j\left[n\right]$ from a mixture of J sources. Up to $J=4$ source utterances were considered. To construct the Elevatogram matrix, $\mathbf{P}$, as described in [5], the total number of quantization levels used was $L=100$ and the number of tiles was $\mathcal{T}\times \mathcal{T}=2\times 2$ to keep the Matlab computation time low. An analysis of the improvement obtained by increasing the tiling parameter is given in [5]. The performance of the proposed techniques was evaluated using relative delays in the range $1\le {\delta }_j\le 200$ samples. The number of equally spaced azimuth positions was $M=1000$ in the range $0\le g\le 1$ for Elevato-AdRess. It was assumed that the difference in relative delay between any two pairs of sources could not be less than $\Delta g<150$ units on the gain axis and that it should be separated by at least 40 samples of relative delay so that the peaks were sufficiently apart, and thus, were distinguishable (Figure 4).

The quality of the separated speech utterances was evaluated using BSS_Eval [102] and PEASS [103] and their intelligibility was measured using Perceptual Evaluation of Speech Quality (PESQ) [104] scores. The Ideal Binary Mask (IBM) is considered the ultimate goal of Computational Auditory Scene Analysis (CASA) [105]. The target source is judged to be present at TF points where it dominates its interference. Consequently, it is preserved. In the regions where the source has a weaker presence than the interference, it is discarded. To evaluate the quality of recovered utterances, the two proposed methods were benchmarked against the IBM (https://github.com/IoSR-Surrey/MatlabToolbox, accessed on 15 April 2025) [106], which is assumed to give the best achievable separation quality. Given a sampling rate of $F_s=16$ kHz, in a noiseless scenario, the two benchmark techniques achieved the following performance. DUET estimates delays of up to 7 samples. AdRess was designed for instantaneous scenarios. This is captured by the poor performance of DUET and AdRess for large relative delays in Figure 5.

Speech enhancement and demixing algorithms employ various analysis and synthesis techniques, in the form of spectral energy and phase manipulation techniques, that introduce distortions, noise, and artifacts, and are typically called filtering errors [107]. In summary, measurements tend to decrease as the number of sources in the mixture increases, primarily due to the higher probability of sources overlapping in the TF bins. The estimate of the target source, ${\widehat{s}}_{\mathrm{j}}\left[n\right]$, can be decomposed into the sum of four separate components,

$${\widehat{s}}_{\mathrm{j}}\left[n\right]=s_{\mathrm{j}}\left[n\right]+e_{\mathrm{j}}^{\mathrm{interference}}\left[n\right]+e_j^{\mathrm{noise}}\left[n\right]+e_{\mathrm{j}}^{\mathrm{artifact}}\left[n\right],$$

(43)

where $s_{\mathrm{j}}\left[n\right]$ is the clean speech utterance [102]. The Source-to-Distortion (SDR) separation quality is defined as

$$\mathrm{SDR}={\displaystyle \frac{\left|\right|s_{\mathrm{j}}\left[n\right]|{|}^2}{\left|\right|e_j^{\mathrm{target}}\left[n\right]+e_{\mathrm{j}}^{\mathrm{interference}}\left[n\right]+e_j^{\mathrm{noise}}\left[n\right]|{|}^2}}.$$

(44)

It gives a measure of how well the target utterance has been recovered. The SIR is defined as

$$\mathrm{SIR}={\displaystyle \frac{\left|\right|s_{\mathrm{j}}\left[n\right]|{|}^2}{\left|\right|e_{\mathrm{j}}^{\mathrm{interference}}\left[n\right]|{|}^2}}.$$

(45)

It computes a measure of how much the energy leaks from the other interfering sources, or how loud they can be heard in the recovered speech utterance. The Source-to-Artifact Ratio (SAR) is given as,

$$\mathrm{SAR}={\displaystyle \frac{\left|\right|s_{\mathrm{j}}\left[n\right]+e_{\mathrm{j}}^{\mathrm{interference}}\left[n\right]+e_{\mathrm{j}}^{\mathrm{noise}}\left[n\right]|{|}^2}{\left|\right|e_{\mathrm{j}}^{\mathrm{artifact}}\left[n\right]|{|}^2}}.$$

(46)

This measure attempts to quantify burbling sounds, which are also referred to as musical-noise, in the reconstructed separated utterances. Figure 5 illustrates that Elevato-DUET and Elevato-AdRess can function effectively with relative delays of up to 200 samples. This capability arises from combining the Elevatogram technique, which was contributed in [5], with the classical DUET [64] and AdRess [65] methods, enhancing their performance in scenarios where the acoustic sources experience significant relative delays. The figure further highlights that Elevato-AdRess outperforms Elevato-DUET in terms of the SDR, SIR, and SAR separation quality metrics.

As shown in Figure 5a, Elevato-AdRess gives better results than Elevato-DUET by a mean reconstruction SDR gain of $2.2$ dB for relative delays of up to 120 samples. Beyond this point, a performance decrease is observed due to an increase in the Mean Absolute Error (MAE) between the true delay, ${\delta }_{\mathrm{j}}$, and its estimated value, ${\widehat{\delta }}_{\mathrm{j}}$, which is calculated by the Elevatogram [5], e.g., $|{\delta }_j-{\widehat{\delta }}_j|$. As can be seen in Figure 5b, the mean reconstruction SIR of Elevato-AdRess exceeds that of Elevato-DUET by $4.3$ dB for a relative delay of ${\delta }_j=100$ samples. Figure 5c shows that the difference in mean reconstruction SAR between the two algorithms is small. It lies within $1.5$ dB for relative delays of up to 120 samples. In general, Elevato-AdRess improves the separation quality, in terms of the SDR, SIR, and SAR, compared with the utterances recovered by Elevato-DUET.

Figure 6 illustrates the boxplots of the measurements obtained for each mixture category. The categories correspond to 2-source, 3-source and 4-source mixtures. The PEASS toolbox [103] estimates the distortion, which is considered to be the error between the estimated source, ${\widehat{s}}_{\mathrm{j}}\left[n\right]$, and the clean speech $s_{\mathbf{j}}\left[n\right]$ and given as

$${\widehat{s}}_{\mathrm{j}}\left[n\right]-s_{\mathrm{j}}\left[n\right]=e_j^{\mathrm{target}}\left[n\right]+e_{\mathrm{j}}^{\mathrm{interference}}\left[n\right]+e_{\mathrm{j}}^{\mathrm{artifact}}\left[n\right].$$

(47)

The target distortion, $e_{\mathrm{j}}^{\mathrm{target}}\left[n\right]$, indicates how many distortions (both attenuation and amplification distortions [108]) are added to the true target source during the filtering process. The measure of interference-related source energies present in the recovered utterance is denoted by $e_{\mathrm{j}}^{\mathrm{interference}}\left[n\right]$ and $e_{\mathrm{j}}^{\mathrm{artifact}}\left[n\right]$ is the measure of musical noise that degrades the demixed speech. Each recovered utterance, ${\widehat{s}}_{\mathrm{j}}\left[n\right]$, is evaluated relative to the reference. The target signal, ${\widehat{s}}_{\mathrm{j}}\left[n\right]$, is on a scale which ranges from 0 to 100 units, with higher scores indicating better quality of separation. As can be seen in Figure 6, Elevato-AdRess achieves a better Overall-Perceptual Score (OPS), Target-Related Perceptual Score (TPS), and Interference-Related Perceptual Score (IPS) performance than Elevato-DUET.

Figure 6a shows that Elevato-AdRess outperforms Elevato-DUET in terms of the average OPS score, with a mean reconstruction improvement of 2 units for mixtures with two sources. This difference becomes more pronounced as the number of speech utterances in the mixtures increases, reaching a difference of 7 units for mixtures with four sources. Figure 6b highlights reduced distortions are experienced in the estimated utterances recovered by Elevato-AdRess. This is reflected by the average TPS scores, where Elevato-AdRess exceeds Elevato-DUET by 8 units in the case of two sources. For mixtures with four sources, the TPS score difference increases to 24 units.

In Figure 6c, reduced energy leakage from interfering sources is observed in the estimates demixed by Elevato-AdRess compared to Elevato-DUET. For a two-source case, Elevato-AdRess outperforms Elevato-DUET by a score of 18 units and the latter lags behind the former by a score of 12 units when four sources are present in the mixture.

The highest PEASS metric achieved by Elevato-AdRess resulted from reduced interference in the recovered utterance. The average IPS score exceeded 85 units and remained relatively stable, even as the number of constituent sources in the mixture increased, in contrast to Elevato-DUET, which showed a notable decline under similar conditions. In terms of artifact-related noise, Figure 6d reveals that Elevato-DUET achieves a higher Artifact-Related Perceptual Score (APS), indicating fewer artifacts compared to Elevato-AdRess. Specifically, Elevato-AdRess is outperformed by Elevato-DUET by an average score of 3 units in the two-source scenario. This difference increases slightly to 5 units in the case of four sources. This result aligns with the higher SNR observed for Elevato-DUET, as shown in Figure 7. A reasonable assumption is that discarding the overlapping frequency bin energies, the approach taken by Elevato-DUET, introduces significant distortion and artifact noise in the separated utterances. SNRs are typically high if they are greater than 5 dB, low if they are less than < dB, and completely unintelligible if they are less than 0 dB according to [3]. In general, the SNRs of the separated speech utterances decrease with an increase in the number of constituent sources in the mixture, where the SNR between the clean utterance, $s_{\mathrm{j}}\left[n\right]$, and the recovered utterance ${\widehat{s}}_{\mathrm{j}}\left[n\right]$, is defined as

$$\begin{array}{c}\hfill \mathrm{SNR}=10{log}_{10}{\displaystyle \frac{\left|\right|s_{\mathrm{j}}\left[n\right]|{|}_2^2}{\left|\right|s_{\mathrm{j}}\left[n\right]-{\widehat{s}}_{\mathrm{j}}\left[n\right]|{|}_2^2}}.\end{array}$$

(48)

In Figure 7, it can be observed that Elevato-DUET improved SNR performance compared to Elevato-AdRess by a measure of $4.9$, $4.5$, and $4.13$ dB for the two, three, and four mixtures cases, respectively.

Higher quality speech-utterance separation does not necessarily mean that high intelligibility is achieved [108,109]. A good speech demixing and enhancement algorithm must preserve intelligibility while improving the quality of separation. To evaluate intelligibility, the PESQ Mean Opinion Score (MOS) [104] intelligibility assessment is used. These scores range from $-0.5$ to $4.5$ units with scores below 1 unit indicating unintelligible speech and higher scores reflecting better intelligibility. Comparing an utterance with itself gives a MOS score of ≥4 units. Figure 8 quantifies the intelligibility of the demixed speech utterances. For a mixture of two sources, $J=2$, Elevato-AdRess achieves a NarrowBand (NB) MOS score of $1.4$ units higher than Elevato-DUET. This difference increases to $1.73$ units in a four-source scenario, $J=4$, as shown in Figure 8 (left). In this scenario, Elevato-AdRess achieves better performance than the IBM by $0.3$ units. Figure 8 (middle) illustrates that in terms of the Listening Quality Objective (LQO) score, Elevato-AdRess gives better performance than Elevato-DUET and IBM by $0.57$ units for the two-source case. This improvement rises to $1.1$ units when there are $J=4$ sources in the mixtures. Finally, Figure 8 (right) highlights the WideBand (WB) MOS scores, which extend the NB evaluation to account for broader frequency ranges, further demonstrating the superior performance of Elevato-AdRess.

Note that the standard separation quality achieved by Elevato-AdRess is largely preserved, as the number of constituent sources in the speech mixtures increases from two to four. This demonstrates the robustness of Elevato-AdRess, as it remains relatively unaffected by the increase in number of sources in the mixture.

6. Discussion and Future Work

This paper introduced a reformulation of the AdRess demixing technique, an extension of the classical DUET algorithm. AdRess was developed for the pan-pot mixing model to address target speaker recovery in anechoic, competing speaker scenarios characterized by relative attenuations and delays. A stereo mixture provides spatial cues that help identify a signal propagating from a specific direction. Using this information, we proposed the use of a remote microphone to isolate the speaker of interest and directed the separated audio to a smart HA, enhancing speech accessibility for individuals with hearing impairments.

The main contribution of this paper lies in extending two source demixing algorithms—DUET and AdRess—by incorporating a large relative delay estimation technique called the Elevatogram, which was introduced in [5], with the derived TF masks. The masks developed used both spectral and spatial information and can function in anechoic scenarios. The two proposed techniques, Elevato-DUET and Elevato-AdRess, outperformed the benchmark techniques in scenarios where the speech source suffers large relative delays. In the experiments, the speech utterances were mixed in an anechoic model with different attenuation and relative delays. Care was taken to ensure distinct spatial signatures—relative attenuations and delays—for each source, minimizing proximity effects that could hinder separation performance. In both proposed TF-masks, the spectral cue depends on the spatial cue. The step concerning accurate delay subtraction helped us to localize the spectral energy of the target source on the frequency–azimuth plane. It is essential to estimate the relative delay accurately for the proposed techniques to function. Inaccurate spatial cues will localize the spectral energies at erroneous positions on the gain axis in Figure 3a. In Elevato-DUET, it is important to maintain the quality of the TF mask. Unlike Elevato-AdRess, which focuses solely on localizing the IID cues of the target speaker, Elevato-DUET requires estimating the attenuation–delay pairs for all constituent sources. The TF mask, in Equation (37), is then computed by iterating over these pairs to derive the best mask to isolate the target source. As demonstrated in [9], the failure to find a good mask results in lower separation quality. In short, these results underline the importance of accurate relative delay estimation, when the separation between sensors is large in AL environments, and provide additional evidence that the Elevatorgram [5] provides accurate large relative delay estimates.

In summary, in terms of speech separation quality, as measured by BSS_Eval metrics (SDR, SAR, and SIR) and PEASS scores (OPS, TPS, IPS, and APS), as well as intelligibility quality assessed by PESQ, Elevato-AdRess demonstrated superior performance compared to Elevato-DUET. The Elevato-AdRess technique focuses on isolating the components of the frequency–azimuth matrix, where the IID spectral cues of the target speaker are localized under the constraint that discrete frequencies must be non-overlapping. TF-bins where the spectral energies overlap on the frequency–azimuth plane, in Figure 3c, are excluded during the extraction process, leading to a loss of information that significantly introduces distortion that affects the SDR and SNR scores of recovered speech utterances. Although this approach increases the computation time in MATLAB due to the localization of spectral energies across $M=1000$, discrete azimuth levels, it offers the advantage of reducing interference in the recovered utterance compared to Elevato-DUET, but this comes at the cost of a significant increase in MATLAB computation time. Elevato-DUET is computationally efficient compared to Elevato-AdRess. One of the key motivations behind the interest in Elevato-AdRess is that its computational time can be reduced, since it can be computed in parallel, making it suitable for real-time implementation. Computing the WDO measure in Equation (33) and the PSR measure in Equation (40), for Elevato-AdRess kind of masking is a challenge. This is because AdRess-based masking uses a brute-force method of explicitly localizing the non-overlapping spectral energies of a speech utterance on the frequency–azimuth plane. It is not the matrix-mask, $\mathbf{M}[k,\tau ]\in \{1,0\}$, in the traditional sense as that of DUET. Elevato-AdRess and Elevato-DUET perform well in noiseless anechoic scenarios. Future work will focus on investigating how the quality of separation in Elevato-DUET deteriorates with an increasing presence of overlapping TF bins in the mixture, by increasing the number of speakers in the AL environment. The Elevato-AdRess and Elevato-DUET algorithms demonstrate improvements in intelligibility by effectively addressing the phase wraparound issue. However, aggressive source separation may introduce audible artifacts and spectral distortions. We will explore adaptive post-processing techniques to minimize these distortions, thereby enhancing perceived speech quality. For example, in the context of music source separation, once the AdRess algorithm has been applied to determine the stems in the original mixture—individual component tracks of a mixed audio recording—some of the mixture is added back into the demixed stems to improve the perceptual quality of the resulting sources.

7. Conclusions

A source demixing technique originally designed to separate music sources mixed using the pan-pot model was reformulated to operate in an anechoic scenario characterized by relative attenuations and large delays. The primary objective was to isolate a target speaker of interest using multiple microphones from a mixture of concurrently active voices in a competing acoustic environment, and to transmit the isolated speech to a hearing-aid worn by a patient with hearing impairment in an AL environment. A significant challenge in real-world scenarios is the phase wraparound issue, which arises when the distance between a pair of microphones exceeds the smallest allowable wavelength of the acoustic wave. This causes severe corruption of higher frequencies, rendering traditional separation algorithms ineffective. Demixing algorithms function as filtering techniques involving both analysis and synthesis processes, which may introduce distortions and artifacts into the synthesized utterance.

Building on the work in [4,5,9], this paper proposed a speech IS scheme utilizing a remote microphone that constituted a near-field microphone setup. The RM-aided IS scheme was designed to enhance the intelligibility of a distant target speaker. Specifically, this paper investigated the feasibility of extending two classical source demixing algorithms, DUET and AdRess, by combining a large delay estimation technique known as the Elevatogram [5]. Generally, the challenge of phase wraparound hinders accurate localization of a source in space. The Elevatogram technique in [5] overcomes this challenge. Both Elevato-DUET and Elevato-AdRess are spatio-spectral masking techniques in which the IID cue depends on the accurate estimation of the IPD cue, making the two cues interrelated. These enhanced versions, named Elevato-AdRess and Elevato-DUET, are designed to operate effectively in scenarios with substantial relative delays of up to 200 samples. This advancement holds significant potential for applications in hearing-assisted living scenarios. Through the derived masks of these algorithms, we demonstrated their capability to perform robustly in large relative delay conditions. In conclusion, the Elevato-AdRess demonstrated superior performance compared to Elevato-DUET in terms of speech separation quality, exhibiting fewer distortions, reduced interference, and enhanced intelligibility for human listeners. It was observed that the Elevato-AdRess technique, by discarding overlapping frequencies, introduced significant artifact noise into the recovered utterance, resulting in a reduced SNR for the extracted source.

Finally, the correlation between the BSS_Eval and PEASS separation quality metrics was demonstrated. Specifically, the SDR exhibited a strong correlation with the OPS, which evaluates the quality of the separated utterance. Similarly, the SIR showed a significant correlation with the interference-related IPS metric. The poor quality of separation demonstrated lower scores in both types of objective measures. The SAR of BSS_Eval demonstrated little correlation with the APS of PEASS. The Elevato-AdRess demonstrated better interference reduction in the recovered signal than Elevato-DUET. It remained stable with increases in the number of constituent sources in the mixture. An accurate estimation of the relative delay was crucial for the operation of the Elevato-AdRess because erroneous estimates rendered the application of the mask unsuccessful. The MAE, $|{\delta }_j-{\widehat{\delta }}_j|$, was also crucial. An estimation error of more than 1 sample rendered the Elevato-AdRess ineffective. From the perspective of human auditory perception, Elevato-AdRess demonstrated better performance in terms of the PESQ MOS intelligibility score compared to Elevato-DUET.