Particle-Velocity-Based Mixed-Source Sound Field Translation for Binaural Reproduction

Abstract

Following the rise of virtual reality is a demand for sound field reproduction techniques that allow the user to interact and move within acoustic reproductions with six-degrees-of-freedom. To this end, a mixed-source model of near-field and far-field virtual sources has been introduced to improve the performance of sound field translation in binaural reproductions of spatial audio recordings. The previous works, however, expand the sound field in terms of the mixed sources based on sound pressure. In this paper, we develop a new mixed-source expansion based on particle velocity, which contributes to more precise reconstruction of the interaural phase difference and, therefore, contributes to improved human perception of sound localization. We represent particle velocity over space using velocity coefficients in the spherical harmonic domain, and the driving signals of the virtual mixed-sources are estimated by constructing cost functions to optimize the velocity coefficients. Compared to the state-of-the-art method, sound-pressure-based mixed-source expansion, we show through numerical simulations that the proposed particle-velocity-based mixed-source expansion has better reconstruction performance in sparse solutions, allowing for sound field translation with better perceptual immersion over a larger space. Finally, we perceptually validate the proposed method through a Multiple Stimulus with Hidden Reference and Anchor (MUSHRA) experiment for a single source scenario. The experimental results support the better perceptual immersion of the proposed method.

1. Introduction

Sound field translation plays an important role in binaural reproduction systems [1], which enables a listener to move about an acoustic space with a sustained perceptual immersion. A typical example of its application is in virtual-reality reproductions of real-world experiences, where the listener is allowed to virtually explore by moving their body/head and they can experience a spatially accurate perception that is the same as in the real-world space [2]. In this paper, we propose a particle-velocity-based mixed-source expansion that can provide a wide range of sound field translation with immersive human perception.

There have been various studies of sound field translation that allow for a listener to interact and move within a sound field reproduction, and they can be divided into two main categories: interpolation-based techniques [3] and extrapolation-based techniques [4]. The interpolation-based techniques aim to interpolate the sound field to listeners during reproduction using a grid of higher-order microphones (inpractice, higher-order microphones are designed from several pressure microphones equally spaced on a solid sphere) over space [5]. For example, a simple interpolation-based technique is shown in [6], where the room impulse response is interpolated to enable real-time auralizations with head rotation. The interpolation-based techniques usually suffer from a significant localization error [5,7] and comb-filtering spectral distortions [8], in particular when the sound source is nearer to one microphone than to another. Although the comb-filtering spectral distortions can be avoided using a parametric Ambisonics room impulse response interpolation system [9], it requires extra effort to compensate for varying interference. In addition, the interpolation assumes multiple distributed microphones or even arrays of microphones, which is a significant hardware limitation in many Augmented Reality/Virtual Reality (AR/VR) applications [3,10]. To enhance the performance of perceptual source localization for displaced listeners, a modified method is proposed, where the directional components of the microphone nearest to the listener are emphasized [11]. This method also interpolates the sound field to listeners and is subject to the fundamental limitation. Methods that alleviate the fundamental limitation have been investigated; however, they usually require additional source localization and separation of direct sound field components [12,13,14]. On the other hand, the extrapolation-based techniques spherically extend the translation from the recording microphone to the nearest source, which can be achieved by Ambisonics [15,16], discrete virtual sources [17], plane-wave expansion [18,19,20], and the equivalent source method [21]. A comparison between different extrapolation-based techniques is given in [22]. To improve the performance of translation and source localization, especially for off-center positions, optimized decoding methods such as max-$r_{\mathrm{E}}$ (meaning max energy vector magnitude) decoding [23] and near-field-compensated decoding [24] have been proposed. However, most of the extrapolated-based techniques are limited by the order of the recordings, which is determined by both the upper frequency band and the microphone radius [25,26]. Consequently, for a low-order recording, the range of translation is significantly limited. Attempts to move beyond this limited range, even after extrapolation, result in spectral distortions [27,28,29], degraded source localization [22,30], and a poor perceptual listening experience.

Parametric models have been shown to provide an efficient way to describe sound scenes [14,31,32], which can be used in binaural reproduction. To overcome the limitation of translation range discussed above, a parametric model is built to describe first-order Ambisonics recordings for translation with known information of a listener’s position and source distances [33]. A similar approach was described in [34], which extends to support arbitrary listener movement; however, with multiple source directions and distances. In [35], the authors improve the spatial localization accuracy by additional spatial information obtained from multi-perspective Ambisonics recordings. Higher order Ambisonics signals are used in [36], which translates the sound field based on multi-directional decomposition that estimates source distances. Another parametric decomposition from a higher order Ambisonics signal has been proposed in [37], offering excellent localization performance even for strong spatial displacements. However, it also requires a prior knowledge of source distance information and an additional analysis stage of direction of arrival estimation. The sound field translation with very few measurements in a room is investigated in [38]. Recently, an alternative secondary source model for extrapolated virtual reproduction has been developed, where it sparsely expands the recording into the equivalent virtual environment using a mix of near-field and far-field virtual sources [39]. It has been shown that this method can relax the limitation and offer an improved perceptual experience through perceptual listening tests [40]. However, this method expands the sound field based on sound pressure, which is not an acoustic quantity directly linked with human perception of sound localization.

Since Gerzon first developed velocity theory of sound localization based on binarual phase cues for reproducing psychoacoustically optimum sound [41], particle velocity (vector), which is recently controlled in sound field reproduction systems [42,43,44], has been one of the objective metrics predicting sound direction. Thus, having particle velocity as the design criterion contributes to a more precise reconstruction of the interaural phase difference, which is crucial for human perception of sound directions and, therefore, contributes to improved localization of the reconstructed sound, especially for low frequencies (below 700 Hz) [45]. For example, a particle-velocity-based sound field reproduction method is proposed to improve the perception experience for listeners away from the center of the reproduction area [46]. Furthermore, the particle velocity-controlled sound field shows an accurate reproduction of sound intensity in the reproduction region [47,48], where sound intensity is another important acoustic quantity for localization perception; however, it is most appropriate at high frequencies (above 500 Hz) [49,50,51]. In this paper, we propose a new mixed-source expansion based on particle velocity for sound field translation and synthesis. We aim to reproduce a spatial acoustic scene by matching particle velocity of the sound field over a region with a sparse distribution of virtual sound sources placed in both the near and far field. We will show that sparsely expanding the sound field in terms of particle velocity offers a more immersive perceptual experience at the translated positions, especially for source localization.

The remainder of this paper is structured as follows. First, the sound field translation problem is formulated in Section 2. We have recently formulated continuous particle velocity expressions over space in the spherical harmonic domain [44], which contain the directivity information of a sound field over continuous spatial regions. In Section 3, we review the continuous particle velocity theory and develop the particle-velocity-based expansion by exploiting this theory in the mixed-source model. Section 4 translates the sound field and synthesizes binaural signals for translated listeners. In Section 5, we introduce two localization metrics, in addition to the relative pressure error, as evaluation criteria. With these criteria, we demonstrate the reproduction accuracy of the proposed method by comparing it with the state-of-the-art method, sound-pressure-based mixed-source expansion, through numerical simulations. The perceptual validation through a MUSHRA experiement is presented in Section 6. Finally, the work is concluded in Section 7.

2. Problem Formulation

Consider a listener within an acoustic environment, where the listening space of the environment is centered by the origin $\mathit{o}$. The listener is positioned at $\mathit{d}=(r,\theta ,\varphi )$, where $\theta \in [0,\pi ]$ and $\varphi \in [0,2\pi )$ in spherical coordinates, with respect to the origin. Let there be a total of K sound sources outside the listening space, with the $\kappa$th source located at ${\mathit{b}}_{\kappa }$. The free-field binaural sound perceived by the listener can be synthesized by filtering the source signals with the listener’s head-related transfer functions (HRTFs), expressed as

$$P_{\mathrm{L},\mathrm{R}}(\mathit{d},k)=\sum _{\kappa =1}^K{H}_{\mathrm{L},\mathrm{R}}(k,{\mathit{b}}_{\kappa };\mathit{d})s_{\kappa }\left(k\right),$$

(1)

where $P_{\mathrm{L},\mathrm{R}}(\mathit{d},k)$ is the sound pressure at the listener’s left and right ear, $H_{\mathrm{L},\mathrm{R}}(k,{\mathit{b}}_{\kappa };\mathit{d})$ are the HRTFs between the $\kappa$th source and the listener’s ears, $s_{\kappa }\left(k\right)$ is the sound signal of the $\kappa$th source, $k=2\pi f/c$ is the wave number, f is frequency, and c is the speed of sound propagation.

We aim to record and reproduce the auditory experience of (1) for every possible listening position. The acoustic environment can be characterized using a spherical harmonic decomposition of the sound pressure, expressed as

$$P(\mathit{x},k)=\sum _{n=0}^{\infty }\sum _{m=-n}^n{\alpha }_{nm}\left(k\right)j_n\left(k\right|\mathit{x}\left|\right)Y_{nm}\left(\widehat{\mathit{x}}\right),$$

(2)

where $\mathit{x}$ is an arbitrary position in the space, $|·|\equiv r$, $\widehat{·}\equiv (\theta ,\varphi )$, ${\alpha }_{nm}\left(k\right)$ are the pressure coefficients which completely describe the source-free acoustic environment centered about $\mathit{o}$, $j_n(·)$ is the nth order spherical Bessel function of the first kind, and $Y_{nm}\left(\widehat{\mathit{x}}\right)$ is the spherical harmonic of order n and degree m, defined by

$$Y_{nm}\left(\widehat{\mathit{x}}\right)\triangleq \sqrt{\frac{(2n+1)}{4\pi }\frac{(n-m)!}{(n+m)!}}{P}_{nm}(cos\theta )e^{im\varphi },$$

(3)

where $P_{nm}(cos\theta )$ is the associated Legendre function. In practice, the pressure coefficients describing the acoustic environment $\left\{{\alpha }_{nm}\left(k\right)\right\}$ can be estimated from recordings of a higher order microphone or a microphone array [52]. We now consider an Nth order receiver centered at $\mathit{o}$, such as a spherical array [53] or planar array [54], that we use to estimate the acoustic environment for $\left\{{\alpha }_{nm}\left(k\right)\right\}$ up to order N. However, the limited order N introduces a spatial reproduction constraint that the sound field can only be accurately reproduced within the receiver region (i.e., $\left|\mathit{x}\right|\le R$), where R is the radius of the receiver region that is related by $N=\lceil kR\rceil$. This constraint restricts the listener to move within the receiver region (i.e., $\left|\mathit{d}\right|\le R$). If the listener attempts to move beyond the receiver region, they would experience spectral distortions and a loss in perceptual immersion.

The objective of this work is to perceptually enhance the performance of translation to $\left|\mathit{d}\right|>R$ for binaural reproduction from a mode-limited measurement so that a listener can move over a large space in a virtual reconstruction with a sustained perceptual immersion, through the proposed particle-velocity-based mixed-source expansion.

3. Particle-Velocity-Based Mixed-Source Expansion

In this section, we present the theory of particle-velocity-based mixed-source expansion. We first give a brief introduction of the background knowledge involving the proposed theory, i.e., the representation of particle velocity in the spherical harmonic domain [44] in Section 3.1, and the mixed-source model [39] in Section 3.2. The particle-velocity-based mixed-source expansion is developed in Section 3.3.

3.1. Spherical Harmonic Representation of Particle Velocity

Particle velocity is a vector and it can be expressed in spherical coordinates, as

$$\mathit{V}(\mathit{x},k)=V_r(\mathit{x},k)\widehat{\mathit{r}}+V_{\theta }(\mathit{x},k)\widehat{\mathit{\theta }}+V_{\varphi }(\mathit{x},k)\widehat{\mathit{\varphi }},$$

(4)

where $\widehat{\mathit{r}}$, $\widehat{\mathit{\theta }}$, and $\widehat{\mathit{\varphi }}$ are the unit vectors in the r, $\theta$, and $\varphi$ directions, respectively. We have recently formulated continuous particle velocity in the spherical harmonic domain [44], which decomposes each component of the particle velocity vector in terms of spherical harmonic functions, as

$$V_{\mathsf{\Psi }}(\mathit{x},k)=\sum _{p=0}^{Q_{\mathsf{\Psi }}}\sum _{q=-p}^p{X}_{pq}^{\left(\mathsf{\Psi }\right)}(k,|\mathit{x}\left|\right)Y_{pq}\left(\widehat{\mathit{x}}\right);\mathsf{\Psi }\in \{r,\theta ,\varphi \},$$

(5)

where $Q_{\mathsf{\Psi }}$ is the particle velocity truncation order (the spherical harmonic decomposition of particle velocity also contains infinite summation terms, and we truncate it to an explicit order for implementation) in the $\mathsf{\Psi }$ direction, and $X_{pq}^{\left(\mathsf{\Psi }\right)}(k,|\mathit{x}\left|\right)$ denote the velocity coefficients in the $\mathsf{\Psi }$ direction. According to the results in [44], the velocity coefficients are given as

$$X_{pq}^{\left(r\right)}(k,|\mathit{x}\left|\right)=\frac{i}{k{\rho }_{0}c}{\alpha }_{pq}\left(k\right)j_p^{\prime }\left(k\right|\mathit{x}\left|\right),$$

(6)

$$X_{pq}^{\left(\theta \right)}(k,|\mathit{x}\left|\right)=\frac{2\pi i}{k{\rho }_{0}c}{H}_{pq}\sum _{n=\left|q\right|}^N{H}_{nq}{\alpha }_{nq}\left(k\right)\frac{j_n\left(k\right|\mathit{x}\left|\right)}{\left|\mathit{x}\right|}{G}_1,$$

(7)

$$X_{pq}^{\left(\varphi \right)}(k,|\mathit{x}\left|\right)=-\frac{2\pi q}{k{\rho }_{0}c}{H}_{pq}\sum _{n=\left|q\right|}^N{H}_{nq}{\alpha }_{nq}\left(k\right)\frac{j_n\left(k\right|\mathit{x}\left|\right)}{\left|\mathit{x}\right|}{G}_2,$$

(8)

where ${\rho }_0$ is the medium density, $j_p^{\prime }\left(k\right|\mathit{x}\left|\right)$ denotes the derivative of $j_p\left(k\right|\mathit{x}\left|\right)$ with respect to r, and

$$H_{pq}=\frac{{(-1)}^{\frac{q+\left|q\right|}{2}}}{2^{\left|q\right|}\left(\right|q\left|\right)!}\sqrt{\frac{(2p+1)}{4\pi }\frac{(p+|q\left|\right)!}{(p-|q\left|\right)!}},$$

(9)

$$\begin{array}{cc}\hfill G_1& =(n+|q|+1)\mathcal{G}(a,b+\frac{1}{2}-{\delta }_{n+\left|q\right|+1};-{\mu }_1-{\delta }_{n+\left|q\right|+1},-{\mu }_2;{\nu }_1+{\delta }_{n+\left|q\right|},{\nu }_2;{\xi }_1,{\xi }_2)\hfill \\ & -(n+1)\mathcal{G}(a,b+\frac{1}{2};-{\mu }_1,-{\mu }_2;{\nu }_1,{\nu }_2;{\xi }_1,{\xi }_2),\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill G_2& =\mathcal{G}(a,b;-{\mu }_1,-{\mu }_2;{\nu }_1,{\nu }_2;{\xi }_1,{\xi }_2)\hfill \\ & =[{(-1)}^{2b+1}+1]\sum _{j_1=0}^{{\mu }_1}\sum _{j_2=0}^{{\mu }_2}\frac{{(-{\mu }_1)}_{j_1}{\left({\nu }_1\right)}_{j_1}}{{\left({\xi }_1\right)}_{j_1}{j}_1!}\frac{{(-{\mu }_2)}_{j_2}{\left({\nu }_2\right)}_{j_2}}{{\left({\xi }_2\right)}_{j_2}{j}_2!}\frac{B(j_1+j_2+a,b)}{2},\hfill \end{array}$$

(11)

with

$${\delta }_M=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}M\mathrm{is}\mathrm{even}\hfill \\ 0,\hfill & \mathrm{if}M\mathrm{is}\mathrm{odd},\hfill \end{array}\right.$$

(12)

$${\left(a\right)}_j=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}j=0\hfill \\ a(a+1)\dots (a+j-1),\hfill & \mathrm{if}j=1,2,\dots ,\hfill \end{array}\right.$$

(13)

and $a=\left(2\right|q|+1)/2$, $b=(3-{\delta }_{n+\left|q\right|}-{\delta }_{p+\left|q\right|})/2$, ${\mu }_1=-(1-n+|q|-{\delta }_{n+\left|q\right|})/2$, ${\mu }_2=-(1-p+|q|-{\delta }_{p+\left|q\right|})/2$, ${\nu }_1=(2+n+|q|-{\delta }_{n+\left|q\right|})/2$, ${\nu }_2=(2+p+|q|-{\delta }_{p+\left|q\right|})/2$, ${\xi }_1={\xi }_2=\left|q\right|+1$, and $B(·)$ denotes the beta function. The detailed derivations of (6)–(8) can also be found in [44].

Note that for a sound field the truncation order of the velocity field (i.e., $Q_{\mathsf{\Psi }}$) is usually larger than or equal to the truncation order of the pressure field (i.e., N) due to the truncation theorem of the velocity expressions [44]. The velocity coefficients represent the continuous particle velocity distribution of a sound field over space and they are directly determined by the pressure coefficients of the sound field, as well as the radial functions (i.e., the spherical Bessel functions). All other terms in the expressions of the velocity coefficients are either constants or indices-only dependent variables. In other words, we can obtain the continuous particle velocity distribution of a sound field over space directly from the pressure coefficients describing the sound field.

3.2. Mixed-Source Model

Different from the plane wave model [20], which represents a sound field with an equivalent superposition of virtual plane wave sources, the mixed-source model expands the sound field in terms of a mixture of near-field and far-field virtual sources [39]. This mixed-source model can overcome the difficulties of the plane wave expansion in synthesizing the near-field sources.

To blend the model of a near- and far-field virtual source together, we define a normalized point-source to be the mixed-source, defined as

$$P(\mathit{x},k;\mathit{y})=\left|\mathit{y}\right|e^{-ik\left|\mathit{y}\right|}\frac{e^{ik|\mathit{y}-\mathit{x}|}}{4\pi |\mathit{y}-\mathit{x}|},$$

(14)

where $\mathit{y}$ denotes the position of the virtual mixed source. In the spherical harmonic domain, it can be expressed as

$$P(\mathit{x},k;\mathit{y})=\sum _{n=0}^{\infty }\sum _{m=-n}^n\underset{{\beta }_{nm}\left(k\right)}{\underbrace{ik\left|\mathit{y}\right|e^{-ik\left|\mathit{y}\right|}{h}_n\left(k\right|\mathit{y}\left|\right)Y_{nm}^{*}\left(\widehat{\mathit{y}}\right)}}{j}_n\left(k\right|\mathit{x}\left|\right)Y_{nm}\left(\widehat{\mathit{x}}\right),$$

(15)

where $h_n(·)$ is the spherical Hankel function of the first kind, and ${\beta }_{nm}\left(k\right)$ are the pressure coefficients of the mixed source. We note that the mixed source has the property of [26]

$$\underset{\left|\mathit{y}\right|\to +\infty }{lim}P(\mathit{x},k;\mathit{y})=\frac{e^{-ik\widehat{\mathit{y}}·\mathit{x}}}{4\pi },$$

(16)

which shows that the plane-wave expansion can be modeled using the mixed-source expansion placed in the far field.

We construct a virtual equivalent sound field using two concentric spheres of virtual mixed sources. Therefore, the sound field can be expressed, using the mixed-source model, as [39]

$$\begin{array}{cc}\hfill P(\mathit{x},k)& =\int \psi (k,R_N\widehat{\mathit{y}};\mathit{o})\underset{P(\mathit{x},k;R_N\widehat{\mathit{y}})}{\underbrace{R_N{e}^{-ik{R}_N}\frac{e^{ik\left|\right|R_N\widehat{\mathit{y}}-\mathit{x}\left|\right|}}{4\pi \left|\right|R_N\widehat{\mathit{y}}-\mathit{x}\left|\right|}}}d\widehat{\mathit{y}}\hfill \\ & +\int \psi (k,R_F\widehat{\mathit{y}};\mathit{o})\underset{P(\mathit{x},k;R_F\widehat{\mathit{y}})}{\underbrace{R_F{e}^{-ik{R}_F}\frac{e^{ik\left|\right|R_F\widehat{\mathit{y}}-\mathit{x}\left|\right|}}{4\pi \left|\right|R_F\widehat{\mathit{y}}-\mathit{x}\left|\right|}}}d\widehat{\mathit{y}},\hfill \end{array}$$

(17)

where $R_N$ is the radius of the virtual sphere placed in the near field, $R_F$ is the radius of the virtual sphere placed in the far field, $\psi (k,R_N\widehat{\mathit{y}};\mathit{o})$ and $\psi (k,R_F\widehat{\mathit{y}};\mathit{o})$ are the driving functions of the mixed-source distributions as observed at $\mathit{o}$, and $P(\mathit{x},k;R_N\widehat{\mathit{y}})$ and $P(\mathit{x},k;R_F\widehat{\mathit{y}})$ are the mixed sources on the two spheres. Note that the mixed-source model expands the sound field with respect to $\mathit{o}$, which allows us to observe the source distribution at $\mathit{o}$ and estimate the sound at any translated position $\mathit{x}$.

3.3. Particle-Velocity-Based Expansion

In this subsection, we develop the theory of particle-velocity-based mixed-source expansion. The particle velocity is related to the sound pressure by the Euler’s equation [55],

$$\mathit{V}(\mathit{x},k)=\frac{i}{k{\rho }_{0}c}\overrightarrow{\nabla }P(\mathit{x},k),$$

(18)

where $\overrightarrow{\nabla }P(\mathit{x},k)$ denotes the gradient of $P(\mathit{x},k)$ in terms of r, $\theta$, and $\varphi$. Similarly, for the mixed source, we have

$$\mathit{V}(\mathit{x},k;R_X\widehat{\mathit{y}})=\frac{i}{k{\rho }_{0}c}\overrightarrow{\nabla }P(\mathit{x},k;R_X\widehat{\mathit{y}}),$$

(19)

where $R_X\in \{R_N,R_F\}$; therefore, taking the gradient operation on both sides of (17), we can obtain the particle-velocity-based mixed-source expansion

$$\begin{array}{c}\hfill V_{\mathsf{\Psi }}(\mathit{x},k)=\int \psi (k,R_N\widehat{\mathit{y}};\mathit{o})V_{\mathsf{\Psi }}(\mathit{x},k;R_N\widehat{\mathit{y}})d\widehat{\mathit{y}}+\int \psi (k,R_F\widehat{\mathit{y}};\mathit{o})V_{\mathsf{\Psi }}(\mathit{x},k;R_F\widehat{\mathit{y}})d\widehat{\mathit{y}},\end{array}$$

(20)

where $V_{\mathsf{\Psi }}(\mathit{x},k;R_N\widehat{\mathit{y}})$ and $V_{\mathsf{\Psi }}(\mathit{x},k;R_F\widehat{\mathit{y}})$ represent the particle velocity of the virtual sources in the near field and the far field, respectively, in the $\mathsf{\Psi }$ direction.

In practice, the particle-velocity-based mixed-source expansion (20) can be approximated by a finite set of L known virtual sources for each virtual sphere; therefore, (20) can be expressed as

$$\begin{array}{c}\hfill V_{\mathsf{\Psi }}(\mathit{x},k)=\sum _{\ell =1}^L{w}_{\ell }\psi (k,R_N{\widehat{\mathit{y}}}_{\ell };\mathit{o})V_{\mathsf{\Psi }}(\mathit{x},k;R_N{\widehat{\mathit{y}}}_{\ell })+\sum _{\ell =1}^L{w}_{\ell }\psi (k,R_F{\widehat{\mathit{y}}}_{\ell };\mathit{o})V_{\mathsf{\Psi }}(\mathit{x},k;R_F{\widehat{\mathit{y}}}_{\ell }),\end{array}$$

(21)

where $w_{\ell }$ are the the sampling weights for the source distribution, and ${\widehat{\mathit{y}}}_{\ell }$ denotes the position of the ℓth virtual source on each sphere.

Note that the parameters of the mixed-source expansion need to be carefully selected in the implementations. The near-field radius of the virtual sphere can be selected based on the desired maximum translation distance, because the listener cannot move beyond the near-field sphere. As for the far-field sphere, it can be placed anywhere so long as it is in the far-field. Moreover, the selection of the number of the virtual sources per sphere is a trade-off between angular resolution and computation cost. The more virtual sources there are, the more opportunity they have to match the true sound direction of arrival no matter the higher computation cost the system has.

By replacing ${\alpha }_{nm}\left(k\right)$ with ${\beta }_{nm}^{(\ell )}$ (i.e., the pressure coefficients of the ℓth virtual source) in (6)–(8), and then substituting them into (5), the particle velocity due to the ℓth virtual source can be written as

$$V_{\mathsf{\Psi }}(\mathit{x},k;R_X{\widehat{\mathit{y}}}_{\ell })=\sum _{p=0}^{Q_{\mathsf{\Psi }}}\sum _{q=-p}^p{X}_{pq}^{(\mathsf{\Psi },\ell )}(k,|\mathit{x}|;R_X{\widehat{\mathit{y}}}_{\ell })Y_{pq}\left(\widehat{\mathit{x}}\right),$$

(22)

where $X_{pq}^{(\mathsf{\Psi },\ell )}(k,|\mathit{x}\left|\right)$ are the velocity coefficients of the ℓth virtual source. Substituting (5) and (22) into (21), (21) can be simplified as

$$X_{pq}^{\left(\mathsf{\Psi }\right)}(k,|\mathit{x}\left|\right)=\sum _{\ell =1}^{2L}{\omega }_{\ell }\psi (k,R_{\ell }{\widehat{\mathit{y}}}_{\ell };\mathit{o})X_{pq}^{(\mathsf{\Psi },\ell )}(k,|\mathit{x}|;R_{\ell }{\widehat{\mathit{y}}}_{\ell }),$$

(23)

where $R_{\ell }=R_N$ for $\ell \in [1,L]$, and for $\ell \in [L+1,2L]$ we have $R_{\ell }=R_F$, and $w_{\ell }=w_{\ell -L}$.

We denote (23), in matrix form, as

$$\mathit{X}=\mathit{A}\mathit{\psi },$$

(24)

where $\mathit{X}={[{\mathit{X}}_r^T,{\mathit{X}}_{\theta }^T,{\mathit{X}}_{\varphi }^T]}^T$ is a $[{(Q_r+1)}^2+{(Q_{\theta }+1)}^2+{(Q_{\varphi }+1)}^2]$ long vector containing all three components of the particle velocity vector with ${\mathit{X}}_{\mathsf{\Psi }}={[X_{00}^{\left(\mathsf{\Psi }\right)},X_{1(-1)}^{\left(\mathsf{\Psi }\right)},\dots ,X_{Q_{\mathsf{\Psi }}{Q}_{\mathsf{\Psi }}}^{\left(\mathsf{\Psi }\right)}]}^T$, $\mathit{\psi }={[{\psi }_1,\dots ,{\psi }_{\mathcal{L}}]}^T$ is a $\mathcal{L}=2L$ long vector with ${\psi }_{\mathcal{L}}=\psi (k,R_{\mathcal{L}}{\widehat{\mathit{y}}}_{\mathcal{L}};\mathit{o})$, and $\mathit{A}={[{\mathit{A}}_r^T,{\mathit{A}}_{\theta }^T,{\mathit{A}}_{\varphi }^T]}^T$ is the $[{(Q_r+1)}^2+{(Q_{\theta }+1)}^2+{(Q_{\varphi }+1)}^2]$ by $\mathcal{L}$ expansion matrix with

$$\begin{array}{c}\hfill {\mathit{A}}_{\mathsf{\Psi }}=\left[\begin{array}{ccc}{w}_1{X}_{00}^{(\mathsf{\Psi },1)}& \dots & w_{\mathcal{L}}{X}_{00}^{(\mathsf{\Psi },\mathcal{L})}\\ w_1{X}_{1(-1)}^{(\mathsf{\Psi },1)}& \dots & w_{\mathcal{L}}{X}_{1(-1)}^{(\mathsf{\Psi },\mathcal{L})}\\ ⋮& \ddots & ⋮\\ w_1{X}_{Q_{\mathsf{\Psi }}{Q}_{\mathsf{\Psi }}}^{(\mathsf{\Psi },1)}& \dots & w_{\mathcal{L}}{X}_{Q_{\mathsf{\Psi }}{Q}_{\mathsf{\Psi }}}^{(\mathsf{\Psi },\mathcal{L})}\end{array}\right].\end{array}$$

(25)

Note that the dependence is omitted here for notational simplicity. The expansion is now reduced to calculate the driving function of the mixed source distribution $\psi (k,\mathit{y};\mathit{o})$ for a given measured sound field. We introduce two solutions here.

3.3.1. Least Squares Solution

The least squares method is to estimate the driving function $\mathit{\psi }$ so as to minimize the difference between $\mathit{X}$ and $\mathit{A}\mathit{\psi }$, and it can be formulated as

$$\mathit{\psi }=\arg \underset{\mathit{\psi }}{\min }\left|\right|\mathit{A}\mathit{\psi }-{\mathit{X}\left|\right|}_2^2.$$

(26)

This problem can be solved using a Moore–Penrose inverse with Tikhonov regularization [56].

3.3.2. Sparse Solution

It is shown that modeling fewer virtual sources from propagation directions that are similar to the original sound will lead to better perceptual immersion [39]. Therefore, we also construct a sparse constrained objective function, expressed as

$$\mathit{\psi }=\arg \underset{\mathit{\psi }}{\min }\left|\right|\mathit{A}\mathit{\psi }-{\mathit{X}\left|\right|}_2^2+{\lambda \left|\right|\mathit{\psi }\left|\right|}_1,$$

(27)

where $\lambda$ is the parameter controlling the strength of the sparsity constraint for the driving function $\mathit{\psi }$. This sparse linear regression problem can be solved using the least absolute shrinkage and selection operator (LASSO) [57]. Other optimization techniques such as the iteratively reweighted least squares (IRLS) [58] can also be applied to solving this problem. We direct readers to [59] for more details about compressive sensing. Note that the applicability of the above solutions will depend on the condition number of matrix $\mathit{A}$, which is determined by the distribution of the virtual sources. It shows that a well-conditioned $\mathit{A}$ matrix should result from a geometry in which the virtual sources are maximally distributed over space in the literature [26].

4. Sound Field Translation and Synthesis for Binaural Reproduction

As shown in the flowchart of the velocity-based mixed-source sound field translation for binaural reproduction in Figure 1, in this section, we translate the sound field in the spherical harmonic domain, and then synthesize the left and right ear signals at the translated position.

Solving (26) or (27) gives us the reproduced virtual-reality sound field, expressed as

$$\tilde{P}(\mathit{x},k)=\sum _{\ell =1}^{2L}{\omega }_{\ell }\psi (k,R_{\ell }{\widehat{\mathit{y}}}_{\ell };\mathit{o})R_{\ell }{e}^{-ik{R}_{\ell }}\frac{e^{ik\left|\right|R_{\ell }{\widehat{\mathit{y}}}_{\ell }-\mathit{x}\left|\right|}}{4\pi \left|\right|R_{\ell }{\widehat{\mathit{y}}}_{\ell }-\mathit{x}\left|\right|}.$$

(28)

With respect to the translated position $\mathit{d}$, the virtual sources are located at ${\mathit{z}}_{\ell }=R_{\ell }{\widehat{\mathit{y}}}_{\ell }-\mathit{d}$’; therefore, the translated sound field can be written as

$$\tilde{P}(\mathit{x},k;\mathit{d})=\sum _{\ell =1}^{2L}{\omega }_{\ell }\psi (k,R_{\ell }{\widehat{\mathit{y}}}_{\ell };\mathit{o})R_{\ell }{e}^{-ik{R}_{\ell }}\frac{e^{ik\left|\right|{\mathit{z}}_{\ell }-\mathit{x}\left|\right|}}{4\pi \left|\right|{\mathit{z}}_{\ell }-\mathit{x}\left|\right|}.$$

(29)

We decompose (29) in the spherical harmonic domain about the position $\mathit{d}$, and the pressure coefficients of the translated sound field due to the virtual sources at ${\mathit{z}}_{\ell }$ can expressed as

$$\begin{array}{c}\hfill {\gamma }_{vu}(k;\mathit{d})=\sum _{\ell =1}^{2L}{\omega }_{\ell }\psi (k,R_{\ell }{\widehat{\mathit{y}}}_{\ell };\mathit{o})ik{R}_{\ell }{e}^{-ik{R}_{\ell }}{h}_v\left(k\right|{\mathit{z}}_{\ell }\left|\right)Y_{vu}^{*}\left({\widehat{\mathit{z}}}_{\ell }\right),\end{array}$$

(30)

where v and u index the translated harmonics, centered at $\mathit{d}$. The left and right ear signals can be synthesized, in the spherical harmonic domain, directly from the translated pressure coefficients ${\gamma }_{vu}(k;\mathit{d})$ and the coefficients of the listener’s HRTFs, $H_{\mathrm{L},\mathrm{R}}^{vu}\left(k\right)$, with [60,61]

$${\tilde{P}}_{\mathrm{L},\mathrm{R}}(k;\mathit{d})=\sum _{v=0}^{\infty }\sum _{u=-v}^v{\gamma }_{vu}(k;\mathit{d})H_{\mathrm{L},\mathrm{R}}^{vu}\left(k\right).$$

(31)

Note that $H_{\mathrm{L},\mathrm{R}}^{vu}\left(k\right)$ is obtained from the spherical harmonic decomposition of the HRTF $H_{\mathrm{L},\mathrm{R}}(k,\mathit{y};\mathit{d})$.

Alternatively, the left and right ear signals can be obtained by applying the mixed-source driving functions to the HRTFs based on the listener’s translated position, given as

$${\tilde{P}}_{\mathrm{L},\mathrm{R}}(k;\mathit{d})=\sum _{\ell =1}^{2L}{\omega }_{\ell }\psi (k,R_{\ell }{\widehat{\mathit{y}}}_{\ell };\mathit{o})H_{\mathrm{L},\mathrm{R}}(k,{\mathit{y}}_{\ell };\mathit{d}).$$

(32)

We also direct readers to [62,63] for other options of rendering the ear signals at the translated positions.

5. Simulation Study

In this section, we evaluate the performance of the proposed particle-velocity-based mixed-source expansion for binaural reproduction by comparing it with the state-of-the-art method, the sound-pressure-based mixed-source expansion, in simulated acoustic environments. The simulated environments and the evaluation criteria are explained in Section 5.1. The simulation results and discussions are given in Section 5.2 and Section 5.3, respectively.

5.1. Simulation Setup and Criteria

In this simulation, we firstly consider an acoustic environment that contains a single free-field point source with a location of ${\mathit{b}}_1=(1,\pi /3,\pi /4)$. We then consider another scenario where multiple sources exist, which is constructed by adding two more point sources with randomly selected locations of ${\mathit{b}}_2=(0.5,3\pi /4,4\pi /3)$ and ${\mathit{b}}_3=(1.8,\pi /2,\pi /2)$, respectively, in the former acoustic environment. The true sound pressure at an arbitrary position $\mathit{x}$ due to the jth point source is

$$P_j(\mathit{x},k)=\frac{e^{ik\left|\right|{\mathit{b}}_j-\mathit{x}\left|\right|}}{4\pi \left|\right|{\mathit{b}}_j-\mathit{x}\left|\right|},j=\{1,2,3\}.$$

(33)

The total sound pressure due to multiple sources follows the principle of linear superposition. We assume that the receiver can record the sound field up to the 4th order (i.e., $N=4$), and the radius of the receiver region is $R=0.042$ m (We simulate the radius of the EigenMike, which is a popular commercial microphone array [53]). The pressure coefficients describing the acoustic environment $\left\{{\alpha }_{nm}\left(k\right)\right\}$ can be extracted from the receiver’s recording. Here, for simplicity, we use the theoretical pressure coefficients up to the Nth order, expressed as

$${\alpha }_{nm}\left(k\right)=\left\{\begin{array}{c}ik{h}_n\left(k\right|{\mathit{b}}_1\left|\right)Y_{nm}^{*}\left({\widehat{\mathit{b}}}_1\right),n\in [0,N],\mathrm{for}\mathrm{S}1,\hfill \\ \\ \sum _{j=1}^{3}ik{h}_n\left(k\right|{\mathit{b}}_j\left|\right)Y_{nm}^{*}\left({\widehat{\mathit{b}}}_j\right),n\in [0,N],\mathrm{for}\mathrm{S}2,\hfill \end{array}\right.$$

(34)

where S1 denotes the single source scenario and S2 denotes the multiple sources scenario. The mixed-source model consists of two sets of $L=36$ virtual sources distributed on the two spheres, both with the positions arranged on Fliege nodes [64]. The first set is placed in the near field at $R_N=2$ m, and the second set is placed in far field at $R_F=20$ m.

Throughout the simulations, the frequency of the sources is $f=1500$ Hz, except the evaluation in terms of the frequency. The sound speed is $c=343$ m/s and the air density is ${\rho }_0=1.29$ kg/m${}^3$. The truncation orders of the velocity coefficients in the r, $\theta$, $\varphi$ direction are $Q_r=N$, $Q_{\theta }=2N$, and $Q_{\varphi }=2N$, respectively, according to the truncation theorem of the velocity expressions [44]. For the sparse solution, we use a 500 iterations LASSO.

As the first objective performance measure for the reproduction method, we define the mismatch between the reproduced sound field and the true sound field at any point $\mathit{x}$ as

$$ϵ(\mathit{x},k)=\frac{|P(\mathit{x},k)-\tilde{P}{(\mathit{x},k)|}^2}{{\left|P(\mathit{x},k)\right|}^2}\times 100(\%).$$

(35)

A value of 0% means the reproduced sound field is exactly the same as the true sound field at the point. The larger the percentage, the greater the difference between the reproduced sound field and the true sound field at the point. A good perception of sound localization requires accurate reproduction of sound direction. To evaluate the performance of the reproduction of sound direction at the translated position $\mathit{d}$, two localization metrics are introduced [22]. The first one is related to the velocity vector at the translated position, which is given by

$${\mathit{r}}_V(k;\mathit{d})=\mathrm{Re}\left(\frac{{\sum }_{\ell =1}^{\mathcal{L}}{\mathcal{U}}_{\ell }\left(k\right){\widehat{\mathit{z}}}_{\ell }}{{\sum }_{\ell =1}^{\mathcal{L}}{\mathcal{U}}_{\ell }\left(k\right)}\right),$$

(36)

where the subscript V denotes velocity, Re$(·)$ denotes the real part, and ${\mathcal{U}}_{\ell }\left(k\right)$ are the effective source gains (accounting for point-source radiation), expressed as

$${\mathcal{U}}_{\ell }\left(k\right)=\frac{e^{ik\left(\right|{\mathit{z}}_{\ell }|-R_{\ell })}}{|{\mathit{z}}_{\ell }|/R_{\ell }}\psi (k,R_{\ell }{\widehat{\mathit{y}}}_{\ell };\mathit{o}).$$

(37)

${\mathit{r}}_V\left(k\right)$ is used to predict localization at low frequencies (below 700 Hz). For high frequencies (above 500 Hz), the second metric related to the intensity vector is given by

$${\mathit{r}}_E(k;\mathit{d})=\frac{{\sum }_{\ell =1}^{\mathcal{L}}{\left|{\mathcal{U}}_{\ell }\left(k\right)\right|}^2{\widehat{\mathit{z}}}_{\ell }}{{\sum }_{\ell =1}^{\mathcal{L}}{\left|{\mathcal{U}}_{\ell }\left(k\right)\right|}^2},$$

(38)

where the subscript E denotes energy. The directions of ${\mathit{r}}_V(k;\mathit{d})$ and ${\mathit{r}}_E(k;\mathit{d})$ indicate the expected localization direction, and their magnitudes, $|{\mathit{r}}_V(k;\mathit{d})|$ and $|{\mathit{r}}_E(k;\mathit{d})|$, indicate the quality of localization [22]. For the point source at $\mathit{b}$, the directional error ${\zeta }_{V,E}(k;\mathit{d})$ at the translated position $\mathit{d}$ is defined as (The direction error is related to the direction difference in angles between the two unit vectors in (39) by $arccos(1-{\zeta }^2/2)$.)

$${\zeta }_{V,E}(k;\mathit{d})=|\frac{{\mathit{r}}_{V,E}(k;\mathit{d})}{|{\mathit{r}}_{V,E}(k;\mathit{d})|}-\frac{\mathit{b}-\mathit{d}}{|\mathit{b}-\mathit{d}|}|.$$

(39)

Ideally, ${\mathit{r}}_V(k;\mathit{d})$ and ${\mathit{r}}_E(k;\mathit{d})$ should be unit vectors that point to the direction of the point source $\mathit{b}$ (i.e., $|{\mathit{r}}_V(k;\mathit{d})|=|{\mathit{r}}_E(k;\mathit{d})|=1$, and ${\zeta }_{V,E}(k;\mathit{d})=0$).

We also implement the two corresponding solutions (i.e., the closed-form solution and the sparse solution) of the state-of-the-art technique, the sound-pressure-based mixed-source expansion [39,40], for comparison.

5.2. Simulation Results

In this section, we first examine the accuracy of the proposed particle-velocity-based mixed-source expansion to reconstruct the pressure field in the two scenarios, and then we evaluate the localization performance in terms of the frequency and translated position using the metrics in Section 5.1 for the single source scenario.

5.2.1. Reproduced Pressure Field

In terms of the single source scenario, Figure 2 shows the true pressure field calculated by (33), and the recorded pressure field calculated using the theoretical pressure coefficients in (34). The black circle denotes the receiver region. The receiver can record the sound field within the receiver region. However, truncation error degrades the accuracy of the pressure field beyond the receiver region, which is the constraint in Section 2 that we wish to relax through the proposed method. To compare the proposed method with the sound-pressure-based expansion, we first show the results of the sound-pressure-based expansion for the closed-form solution and the sparse solution in Figure 3. We observe that the closed-form solution can guarantee the reproduction accuracy within the receiver region. However, for a position away from the receiver region, the mismatch percentage at the position quickly increases to a value close to 100 %, which is similar (but not exact) to the recorded pressure field and is not good enough for translation. The sparse solution improves the performance and reconstructs the pressure field accurately around the receiver region, which allows for the listener to move to the position exterior to the receiver region. Figure 4 shows the reconstructed pressure field and the error field controlled by the least squares solution and the sparse solution using the particle-velocity-based expansion, and these two solutions have the exactly same pressure field mismatch, which means that the least squares solution of the particle-velocity-based expansion has the same performance as the closed-form solution of the sound-pressure-based expansion. However, for the sparse solution of the particle-velocity-based expansion, it provides a larger area with 0 % pressure field mismatch around the receiver region compared with the sparse solution of the sound-pressure-based expansion, which allows for the listener to translate further away from the original receiver region.

As for the multiple sources scenario, the true pressure field and recorded pressure field are given in Figure 5. The pressure field difference between them also increases gradually (i.e., the value of the mismatch gradually increases from 0% to 100%) as the observation point moves away from the target region due to the truncation error. Similar to the evaluation for the single source scenario, we first show the reconstructed pressure fields controlled by the closed-form solution and the sparse solution based on the sound-pressure-based expansion in the multiple sources scenario. The results with the error fields are shown in Figure 6. It shows that the closed-form solution still can only guarantee accurate reproduction around the target region, whereas the sparse solution can provide greater translation area. However, the area with 0% pressure field mismatch is much smaller than that in the single source scenario. Figure 7 shows the results of the least squares solution and the sparse solution based on the particle-velocity-based expansion. Similarly, the result of the least squares solution under the particle-velocity-based expansion is identical to the result of the closed-form solution under the sound-pressure-based expansion in terms of the pressure field mismatch. Although the sparse solution of particle-velocity-based expansion in the multiple sources scenario can provide a smaller translation area than that in the single source scenario, it still has an improved performance compared to the sparse solution of the sound-pressure-based expansion in the multiple sources scenario.

5.2.2. Frequency

To evaluate the broadband performance of the proposed method on direction reproduction at a translated position, we calculate the directional error ${\zeta }_{V,E}$ and the magnitude of ${\mathit{r}}_{V,E}$ (i.e., $|{\mathit{r}}_{V,E}|$) using the two localization metrics introduced in Section 5.1, with respect to the change in frequency. The translated position is $(0.2,0.3,0)$ m in Cartesian coordinates. Figure 8 shows the results of ${\zeta }_V$ and $|{\mathit{r}}_V|$. As expected, the sparse solution of the particle-velocity-based expansion is better than all the other solutions, especially for low frequencies. The results of ${\zeta }_E$ and $|{\mathit{r}}_E|$ are given in Figure 9. We observe that the sparse solution of the particle-velocity-based expansion has the least directional error, and the $|{\mathit{r}}_E|$ for this solution is closest to 1 among all the evaluated methods. Compared to curves for the other method, the curve for the sparse solution of the particle-velocity-based expansion has little fluctuation in terms of frequency, which means the sparse solution of the particle-velocity-based expansion can provide a uniform performance for broadband signals. In addition, the least squares solution of the particle-velocity-based expansion is the same as the closed-form solution of the sound-pressure-based expansion for the performance in terms of frequency.

5.2.3. Translated Position

We also evaluate the performance of each method when the translated position moves away from the receiver region. Figure 10 and Figure 11 show the results of ${\zeta }_V$ and $|{\mathit{r}}_V|$ at 300 Hz, and the results of ${\zeta }_E$ and $|{\mathit{r}}_E|$ at 1500 Hz, respectively, as a function of translation distance along the positive y-axis. Note that we plot Figure 10 and Figure 11 at different frequencies because ${\mathit{r}}_V$ is a good predictor for low frequencies, whereas ${\mathit{r}}_E$ is a good predictor for high frequencies, as mentioned in Section 5.1. From Figure 10a and Figure 11a, we show that all the methods have little error at the origin where the receiver is positioned; however, the performance worsens as the translation distance increases for all the methods. Compared to the sparse solution of the sound-pressure-based expansion, the sparse solution of the particle-velocity-based expansion has less directional error and this advantage becomes more significant for a further translated position, which leads to a better direction reproduction, especially for the translated position far away from the receiver region. The results of Figure 10b and Figure 11b also indicate that the sparse solution of the particle-velocity-based expansion can provide a better quality of localization for the listener than the sparse solution of the sound-pressure-based expansion. Once again, we notice that the least squares solution and the closed-form solution have same performance, and both are worse than the the sparse solutions of the particle-velocity-based expansion and the sound-pressure-based expansion.

From the above analysis, we conclude that the least squares solution of the particle-velocity-based expansion is identical to the closed-form solution of the sound-pressure-based expansion, whereas the sparse solution of the particle-velocity-based expansion has better overall performance than the sparse solution of the sound-pressure-based expansion.

5.3. Discussion

Based on the above simulation results, we give the following comments:

• The particle-velocity-based expansion is derived directly from the sound-pressure-based expansion by the gradient calculation. Therefore, the least squares solution of the particle-velocity-based expansion is mathematically parallel to the closed-form solution of the sound-pressure-based expansion, both of which aim to reconstruct the original truncated recording by distributing energy throughout all virtual sources and inherit the spatial artifacts caused by the truncated measurement. This leads to poor reproduction outside the receiver region.
• Sparse solutions can improve the performance of reproduction outside the receiver region due to the fact that most sound fields can be reproduced accurately by a single or a few virtual sources, and thus exhibit sparsity in space. Therefore, provided that the desired sound is sparse in space, the sparse solutions can be used to relax the restriction we mentioned in Section 2. We should note that the region of accurate translation in the multiple sources scenario becomes smaller than that in the single source scenario for the sparse solution. In addition, the sparse solution is less applicable to highly reverberant fields where the sparsity does not hold.
• Particle velocity contains the direction information of a sound field. For the sparse solutions, we can achieve more accurate sound field reproduction by controlling particle velocity than sound pressure. In other words, the particle-velocity-based solution outperforms the sound-pressure-based solution for sound field reproduction in the cases where there are a limited number of sources. Furthermore, particle velocity reflects the interaural time difference (ITD); therefore, the velocity vector is directly related to the localization predictor for human perception at low frequencies. Particle velocity is also one of the quantities that determine the sound intensity vector (the localization predictor at high frequencies), which reflects another human localization cue of interaural level difference (ILD). Therefore, the sparse solution of particle-velocity-based expansion is expected to provide an enhanced perceptual immersion for the listener.
• Sparse solutions extrapolate the sound field at the translated positions beyond the receiver region according to the sound field within the receiver region by the mixed-source expansion where we use multiple virtual sources to estimate the original source. Therefore, the more the virtual sources are used, the less error the mixed-source expansion has; however, the higher computation cost the system has. In addition, the particle-velocity-based solutions would be more computationally expensive than the sound-pressure-based solutions due to the the extra spherical harmonic decomposition of particle velocity.

We examine the perceptual advantages of the proposed sparse solution of particle-velocity-based expansion by experiments in the next section.

6. Experimental Verification

In this section, we evaluate the perceptual performance of the proposed method against the state-of-the-art method in a listening experiment. The experimental methodology is introduced in detail in Section 6.1, and then we present the statistical results in Section 6.2.

6.1. Experimental Methodology

To generate the binaural test signals, we construct a virtual experimental environment, where there is a 4th order receiver located at the origin and a free-field point source located at $(1,0,0)$ m, with respect to the origin. The receiver region is a spherical region with a radius of 0.042 m, to represent the spatial abilities of common spherical microphone arrays. We assume the microphone array works ideally, and the theoretical pressure coefficients (34) can be extracted from the ideal microphone recording. The parameters of the mixed-source model are the same as those in Section 5.1. We reproduce the sound image for the listener away from the recording area by extrapolation using the proposed method. The listener faces the positive x-axis, and the listener’s head is aligned vertically with the point source on the x-y plane. We used the HRTFs of the FABIAN head and torso simulator [65] from the HUTUBS dataset [66,67]. The source signal is a clip of dry instrumental music with 10 s duration and its spectrum energy mainly distributes in the frequency band below 8 kHz. We process the signals at a frame size of 4096 with 50% overlap and a 16 kHz sampling frequency. We carry out a MUSHRA [68] experiment to compare four translation methods; therefore, for a translated position there are a total of six binaural test signals in this experiment:

• Reference/Hidden reference: The ground truth, which is obtained directly from filtering the theoretical point source signal with the HRTFs.
• Anchor: Signals of the truncated recording at the origin, which are simulated using the pressure coefficients up to order N in (34). No translation is processed, but is filtered by the HRTFs.
• P-CFS: Signals reconstructed using the closed-form solution of the sound-pressure-based expansion and rendered by the HRTFs.
• V-LSS: Signals reconstructed using the least squares solution of the particle-velocity-based expansion and rendered by the HRTFs.
• P-SS: Signals reconstructed using the sparse solution of the sound-pressure-based expansion and rendered by the HRTFs.
• V-SS: Signals reconstructed using the sparse solution of the particle-velocity-based expansion and rendered by the HRTFs.

We generate the above binaural test signals at six different translated positions outside the receiver region. All the translated positions are on the x-y plane, and they are $(0,1,0)$ m, $(0,-1,0)$ m, $(0.5,0.8,0)$ m, $(0.5,-0.8,0)$ m, $(1,1,0)$ m, and $(1,-1,0)$ m, respectively. In total, the scores of 31 subjects (16 females and 15 males) with normal hearing are collected for this listening experiment. Among the participants there are 14 experienced listeners with an audio research background and 17 common listeners without any audio research background. A random set out of the six sets of test signals is selected for each hearing subject. The test signals are played on a computer and the hearing subjects need to wear headphones to listen to the test signals. We provide the participants with Matlab programs containing the test signals with a user interface, and guide them to perform the experiment using their own computer and headphone through online meetings. The experiment includes two tests, and they are the source localization test and the basic audio quality test, respectively. The order in which the test signals are played is also random for each test. For the source localization test, the hearing subjects are asked to score all the test signals against the reference for the perceived direction of the sound source, whereas the hearing subjects need to score on the spectral distortions and other audible processing artifacts with respect to the reference for the basic audio quality test. During the test, the participants can arbitrarily switch between different test signals, and the duration of the experiment is usually within 20 min. The score ranges from 0 to 100. We disqualify the hearing subjects who score the hidden reference below 90, and their results are removed from the final dataset of scores.

6.2. Experimental Results

After removing the scores of the disqualified subjects, we collect the scores of 30 subjects in total for the source localization test, and 29 scores for the basic audio quality test. Box plots showing the results for both tests are given in Figure 12. On each box, the central short red line indicates the median value, and the edges of the box are the 25th and 75th percentile of the scores. The extremes of the whiskers correspond to $q_3+1.5(q_3-q_1)$ and $q_1-1.5(q_3-q_1)$, where $q_1$ and $q_3$ are the 25th and 75th percentile of the scores, respectively. If a score goes beyond the extremes of the whiskers, it is considered an outlier, indicated by the symbol + in the plots. The symbol ∗ denotes the mean value, and the v-shaped notches refer to the 95% confidence interval. The endpoints of the notches are calculated by $q_2-1.57(q_3-q_1)/\sqrt{N_s}$ and $q_2+1.57(q_3-q_1)/\sqrt{N_s}$, where $q_2$ is the median value and $N_s$ is the number of the scores. The intervals can be used to compare the median values. If two intervals do not overlap, their true medians are different with 95% confidence.

From Figure 12a, we observe that, for the source localization, the closed-form solution of the sound-pressure-based expansion and the least squares solution of the particle-velocity-based mixed-source solution have similar mean values and median values, and their confidence intervals have nearly 100% overlapping. The sparse solutions of both expansions have higher mean values and median values. However, the sparse solution of the particle-velocity-based expansion has a more significant improvement in score and a more centralized score distribution compared to the sparse solution of the sound-pressure-based expansion. Similar results are shown for the basic audio quality test in Figure 12b, where no significant difference is found between the closed-form solution and the least squares solution. For the sparse solutions, the particle-velocity-based expansion is still observed to perform better than the sound-pressure-based expansion; however, both of outperform the closed-form solution and the least squares solution. We also notice that for the sparse solution of the particle-velocity-based expansion the scores in the source localization test are relatively higher than those in the basic audio quality test.

According to the above experimental results, we conclude that in terms of the perceptual criteria of source localization and audio quality the sparse solutions have better performance than the closed-formed solution and the least squares solution, both of which have similar performance. In the sparse solutions, the particle-velocity-based expansion shows an improvement against the sound-pressure-based expansion, especially for the source localization.

6.3. Discussion

Based on the above experimental results, we give the following comments:

• Statistically, the least squares solution of the particle-velocity-based expansion is equivalent to the closed-form solution of the sound-pressure-based expansion, whereas the sparse solution of the the particle-velocity-based expansion provides better perceptual performance than the sparse solution of the sound-pressure-based expansion at a random translated position. The experimental results are consistent with the discussions in Section 5.3.
• Having particle velocity as the design criterion contributes to more precise reconstruction of the interaural phase difference than sound pressure, which is of utmost important for direction perception and therefore, contributes to improved localization of reconstructed sound when there are only a limited number of sources (the sparse solution). In addition to direction perception, audio quality can also be improved by controlling particle velocity for the sparse solution, which may be due to the more accurate reproduction of the sound field.
• There are some limitations for the experimental verification. We only evaluate the single source scenario in the perceptual experiment, where the sound field exhibits sparsity. The perceptual performance may degrade for the multiple source scenario or the highly reverberant sound fields. In addition, the source signal is a single type of music, and the performance of the proposed method in terms of the type of audio remains unexplored. We should note that the equipment used (e.g., headphone and sound card) and the listening environment may differ for different subjects due to the remote experiment.

7. Conclusions

We have proposed a new mixed-source expansion, which exploits particle velocity to enhance the performance of sound field translation for binaural reproduction. We describe spatial particle velocity using the velocity coefficients in the spherical harmonic domain, and expand the sound field in terms of the velocity coefficients. We compare the proposed method with the state-of-the-art expansion based on sound pressure. The simulation results show that the least squares solution of the particle-velocity-based expansion is parallel to the closed-form solution of the sound-pressure-based expansion, whereas the sparse solution of the particle-velocity-based expansion can reproduce the sound field over a larger area with less error, and has more accurate direction reproduction at the translated positions than the sparse solution of the sound-pressure-based expansion. Finally, the results of the MUSHRA experiment verify that the particle-velocity-based expansion outperforms the sound-pressure-based expansion for the sparse solution at the translated positions in terms of both source localization and audio quality for the single source scenario. The improvement for source localization is greater than audio quality.