Adaptive Filtering for Multi-Track Audio Based on Time–Frequency-Based Masking Detection

Abstract

There is a growing need to facilitate the production of recorded music as independent musicians are now key in preserving the broader cultural roles of music. A critical component of the production of music is multitrack mixing, a time-consuming task aimed at, among other things, reducing spectral masking and enhancing clarity. Traditionally, this is achieved by skilled mixing engineers relying on their judgment. In this work, we present an adaptive filtering method based on a novel masking detection scheme capable of identifying masking contributions, including temporal interchangeability between the masker and maskee. This information is then systematically used to design and apply filters. We implement our methods on multitrack music to improve the quality of the raw mix.

1. Introduction

With the advent of the internet and advances in electronic and computing technologies, creating music is becoming more accessible than ever before. In one phrase, music production is now available to the masses [1]. It has become much easier for independent artists to access recording tools, software, and learning resources. Therefore, it is also easier for independent musicians to distribute music and gain recognition without a record label [2]. However, independent musicians are also not being drawn into the mainstream as often, making mainstream music formulaic and overly commercialised. Consequently, music is in danger of losing some of the cultural aspects that have previously made it significant [3].

Despite advances in technology, independent musicians still face challenges in achieving a professional-quality product. One of these challenges is multitrack mixing, which involves applying multiple processes (panning, equalisation, compression, gain, etc.) for blending multiple audio tracks in a digital audio workstation (DAW) [4]. DAWs contain a very large library of tools available to perform a mix. For instance, compressors [5], equalisers [6,7,8], reverberators [9], dereverberators [10], pitch correction [11], and various other effects [12]. Achieving a professional quality mixing still relies on highly experienced mix engineers, and while DAWs have become more affordable and sophisticated, they are still by no means easy to use for someone without training [13]. There are multiple works that address the challenges of mixing for the untrained user. For instance, using machine learning to predict the values of predefined features that will generate a desired outcome [14,15] or studying the methodologies of accomplished mixing engineers [16] or their practices in equalisation [17].

One of the key problems in audio multitrack mixing is spectral masking [18]. This is the inability to discern two sounds if they overlap in time and frequency. Highly developed ear skills are required to remove masking in modern DAWs as they still rely on the user to listen and identify which frequencies need adjustment at which times. Several solutions to the challenge of detecting and removing masking have been proposed. These solutions often use cross-adaptive mechanisms to analyse the influence of a track over the others in order to apply equalisation filters [19]. Features for the analysis, such as intensity at sub-bands [20], or loudness loss [21] are often predefined by the user and these might not be suitable for every piece. Panning has also been used as a means to reduce masking under a similar analysing scheme [22]. These methods might have some setbacks. For instance, relying on the user to predetermine spectral zones or have fixed predefined spectral bands, and might overestimate/underestimate masking due to the omission of the interchangeability of masker and maskee. Equalisation and panning are probably the most effective mechanisms to reduce masking; however, new methods based on artificial intelligence promise not only to reduce masking but to apply all forms of processing in one package [5]. These artificial intelligence methods are still in their infancy and might remove the creative aspects of performing a mix.

There are also a number of commercial software tools that aid in achieving a high-quality mix without conventional training. For instance, soundtheory’s gullfoss [23] uses equalisation to improve quality, the oeksound’s, soothe [24] suppresses prominent frequencies and the Master the mix’s fuser [25] compares characteristics in the working mix with a reference track and provides guidance on how to achieve the same characteristics. The developers of such tools might not reveal how their technology works, and without this understanding, it can be challenging for users to apply these tools correctly. At times, this can be exasperated by the use of artificial intelligence approaches [26,27] where the underlying mechanisms of the algorithms are not clear to the developer either. Mixing remains one of the most challenging aspects of the production of music. Algorithms dedicated to supporting this process are still developing [28], leaving ample room for advancements to fully leverage the accessibility to computing power.

In this article, we propose a simple masking detection method that systematically compares each track with high temporal and spectral resolutions, incorporates the interchangeability between masker and maskee and uses simple user-defined parameters to produce an intricate equalisation filter to improve the clarity of a mix.

2. Evaluation of Masking and Filtering

Our method first calculates the magnitude of the short-time Fourier transforms T of each track:

$$T(t,f)=\left|F\right\{T_r\left(t_0\right)\ast w(t_0-t)\left\}\right|$$

(1)

where $T_r$ is the raw track, $t_0$ the temporal dimension, w the window function, and t the time offset of the window function. The purpose of the window function is to suppress the signal outside of the region of interest to provide resolution in the time domain. In this case, we have used a Hann window [29] of 180 ms in length and spaced by 90 ms. This is a compromise between temporal and spectral resolutions, which enables resolving individual notes (see Figure 1a,b).

We propose Equation (2) to obtain an array of values that identify where in time and frequency masking is occurring, at which severity and in which polarity (i.e., differentiating masker and maskee). We define a masking matrix $M^{\prime }(f,t)$ by:

$$M_{12}^{\prime }(t,f)={\displaystyle \frac{(T_1(t,f)\ast T_2(t,f))(T_1(t,f)-T_2(t,f))}{{(mean\left(T_1(t,f)\right)+mean\left(T_2(t,f)\right))}^3}}$$

(2)

where $T_1$ and $T_2$ are the magnitude of short-time Fourier transforms of track 1 and track 2, respectively. The expression $mean\left(\right)$ indicates a 2D average (scalar). This process creates a matrix of masking $M_{12}^{\prime }(t,f)$ between specific tracks, 1 and 2 (see Figure 1c). The equation consists of three parts:

• $(T_1(t,f)\ast T_2(t,f))$—this term evaluates how relevant it is to calculate masking at this specific time and frequency. For instance, if the amplitude of one track is close to zero, then masking would be small.
• $(T_1(t,f)-T_2(t,f))$—this term evaluates the relative magnitude and polarity of masking. Positive values for $T_1$ indicate this is the masker while negative indicates it is the maskee. This is an important term to differentiate the masker from the maskee at different times.
• ${(mean\left(T_1(t,f)\right)+mean\left(T_2(t,f)\right))}^3$—this is a normalisation term to ensure scale invariance that the same results will be provided if the same non-zero dB gain is applied to both tracks.

The two-dimensional masking matrix $M^{\prime }(f,t)$ is then reduced into a one-dimensional form where all the temporal information is summed while the frequency space is preserved:

$$M_{12}^{\prime }\left(f\right)=\sum _{t=0}^{t=N_t}{M}_{12}^{\prime }(t,f)$$

(3)

where t is the time position of the window function and $N_t$ is the total number of time samples (see Figure 2a). Even though the temporal information is compressed, this only occurs after the calculation of masking takes place. Therefore, the frequencies that contribute to $M^{\prime }\left(f\right)$ are only those that overlap in both time and frequency. This is key to preventing filtering elements of the signals whose components do not exist at the same time in the tracks. After the data are reduced to a single dimension, the positive and negative parts, which correspond to the masking of T1 over T2 (T1 is the masker) and vice versa (T1 is the masquee), are separated. Therefore, $M_{12}^{\prime }\left(f\right)$ becomes:

$$M_{12}\left(f\right)=\left\{\begin{array}{cc}{M}_{12}^{\prime }\left(f\right),\hfill & \mathrm{if}{M}_{12}^{\prime }\left(f\right)>0\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(4)

$$M_{21}\left(f\right)=\left\{\begin{array}{cc}|M_{21}^{\prime }\left(f\right)|,\hfill & \mathrm{if}{M}_{12}^{\prime }\left(f\right)<0\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(5)

where $M_{12}\left(f\right)$ and $M_{21}\left(f\right)$ are the contributions to masking from the masker and the maskee, respectively (see Figure 2b,c). After calculating all contributions to masking from each track to the others (for an arbitrary number of tracks), these are summed and normalised:

$$\begin{array}{cc}\hfill M_{T1}\left(f\right)& ={\displaystyle \frac{M_{1,2}\left(f\right)+M_{1,3}\left(f\right)+...M_{1,N}\left(f\right)}{M_{0T1}}}\hfill \\ \hfill M_{T2}\left(f\right)& ={\displaystyle \frac{M_{2,1}\left(f\right)+M_{2,3}\left(f\right)+...M_{2,N}\left(f\right)}{M_{0T2}}}\hfill \\ \hfill ⋮& \\ \hfill M_{TN}\left(f\right)& ={\displaystyle \frac{M_{N,1}\left(f\right),M_{N,2}\left(f\right)...M_{N,(N-1)}\left(f\right)}{M_{0TN}}}\hfill \end{array}$$

(6)

where N is the number of tracks and $M_{0T1},M_{0T2},...M_{0TN}$ are the normalise-to-unity factors calculated as the maximum value of the sum of the numerators for each track. For example,

$$\begin{array}{c}\hfill M_{0T1}=max\left\{M_{1,2}\left(f\right)+M_{1,3}\left(f\right)+...+M_{1,N}\left(f\right)\right\}.\end{array}$$

(7)

Equation (6) effectively normalises all the masking contributions for each track to others to unity in order to generate the final masking vectors ($M_{T1}...M_{TN}$, see Figure 2d). For an arbitrary number of tracks N, Equation (2) is applied for each pair (see Section 3). The total number of possible combinations between tracks is given by $N\ast (N-1)$ because tracks are only grouped in pairs, and not with themselves (i.e., not $M_{1,1}\left(f\right)$). For stereo tracks, the process is similar, treating left and right channels as independent tracks but not comparing opposite sides of the panning. Mono tracks then can be treated as stereo tracks with identical sides.

Filter Design

The masking vectors $M_{T1}\left(f\right)$, $M_{T2}\left(f\right)$… $M_{TN}\left(f\right)$ obtained from Equation (6) represent the masking contributions of each track to others (see Figure 2d) and so provide then the basis for the reduction of masking by filtering. To design the filters, the most prominent peaks of each masking vector are found by comparing every sample to the two immediate neighbours to detect local maxima (see blue starts on Figure 3a). Then, an adaptive threshold [30] is implemented with a fixed lower limit (see Figure 3b). Applying the threshold reduced the number of peaks originally found by comparing neighbours. This is particularly more noticeable for higher frequencies (see Figure 3c). The central frequency and width of each of the remaining peaks are then used to generate a parametric equalisation filter [31] with the gain of each band proportional to each peak so the relative ratios of masking at different frequencies are preserved (see Figure 3d). The filter is normalised to have a maximum gain of 0 dB and a user-defined minimum gain (for instance, −3 dB in Figure 3d). This process is performed for all masking vectors $M_T\left(f\right)$ and the generated filters are applied to their correspondent tracks in the Fourier space. Finally, the tracks are added up and then this raw mix is normalised to unity (rough master) for evaluation.

3. Results

To demonstrate our methodology, we employ a multitrack from the Mixing Secrets library [13]. The song is Flesh and Bone by Wesley Morgan. The song contains four tracks in the mix (bass, accordion, vocals, and guitar). Tracks with multiple microphone sources were premixed to ensure one track per instrument. The masking matrices for Flesh and Bone obtained using Equation (2) are shown in Figure 4. The most prominent masking comes from the bass to all other tracks (a–c, red); nevertheless, there is masking across all tracks at different extents. Based on these, and using Equations (3)–(6), an attenuating filter was generated for each track (see Figure 5). The filters have multiple peaks attenuating at different bands, except for guitar and vocals at approximately 165 Hz (E4, yellow and dashed purple). Each filter was applied to its correspondent track and then the tracks were averaged with their original levels (rough mix) and normalised to unity (rough master). The gain of each filter is user-defined and set as −8, −6, −3, and −3 dB for bass, accordion, vocals, and guitar, respectively.

For clarity, Figure 6 shows a schematic of how the four tracks were combined, separated, and recombined to achieve the masking vectors required for filter design. In essence, all possible combinations of tracks are used to generate the initial masking vectors using Equations (2) and (3) ($M^{\prime }\left(f\right)$), then masker and maskee are separated (Equations (4)–(5)), averaged, and normalised (Equation (6)), resulting in ($M{\left(f\right)}_T$) as the basis for designing a filter. The number of tracks in this example is four and can be easily expanded to any number.

Figure 7 shows the individual tracks before and after filtering. The changes in the spectrum match the response of the filter presented in Figure 5. Tracks 1 and 2, (bass and accordion, see Figure 7a–d) are modified significantly as the selected gain was aggressive (−8 and −6 dB, respectively). Tracks 3 and 4 (vocals and guitar, see Figure 7e–h) are modified subtly, yet changes in the frequency and time domains are also visible.

4. Discussion

In performing a subjective comparison of the original and filtered mixes for Flesh and Bone (see Supplementary Data), we observe a clear change in the characteristics of the mix. Compared to the original, the filtered mix exhibits less muddiness, particularly on the accordion and vocals, and the levels appear more balanced; the dominance of the accordion in the mix is reduced while the guitar becomes more audible. In our opinion, this is a positive demonstration of our method, considering that there was no adjusting of the gain for each track and no prior information. We have included a second track, Nearly There by Jesse Joy, from the same repository, where the clarity of vocals and guitar are also improved (see Supplementary Data). Besides the gain for each track, further user-defined parameters can be included such as the prominence of peaks of the adaptive threshold and the noise floor adjustment. Modifying these parameters will increase the contribution of the resulting filter, incrementing the number of peaks at higher frequencies.

Finding these intricate filter responses by ear requires a significant amount of practice and training and is by no means an easy task. This is particularly relevant for acoustic or electric instruments recorded using a microphone out of a speaker cabinet (i.e., electric guitar or electric organ). Therefore, there is a benefit in utilising our method, which simplifies the process by reducing the number of parameters in play during mixing.

Our method can be improved by preserving the two-dimensional information and filtering in the STFT space. However, this approach might need significantly more computational resources. This method might not be well suited when mixing different sources of the same instrument as these will have very similar spectral contents.

We envision that our method could be turned into a digital plug-in that can be used as either a pre-processing step or a method to visualise masking and generate editable filters as a mean for musicians untrained in multitrack mixing to improve the quality of their self-produced music.

5. Conclusions

We have presented a novel methodology for the characterisation and reduction of masking in multitrack audio recordings. Our method is an adaptive filtering that systematically compares all tracks of multitrack audio to find the magnitude and polarity of masking in time and frequency. This reduces over-filtering by preventing the contribution of frequencies that overlap only in frequency and not time. Then, this information is used to design filters for each track whose gain can be adjusted. In doing so, we have demonstrated that the resulting mixed audio exhibits a subjective improvement in the quality of the mix (see Supplementary Data). We believe this method can be useful to musicians not trained in audio engineering, to achieve better quality self-produced music.