Next Article in Journal
A Comparative Study on the Laser Welding of Ti6Al4V Alloy Sheets in Flat and Horizontal Positions
Next Article in Special Issue
Late Reverberation Synthesis Using Filtered Velvet Noise
Previous Article in Journal
Influence of Welding Parameters on the Weld Pool Dimensions and Shape in a TIG Configuration
Previous Article in Special Issue
The Reduction of Vertical Interchannel Crosstalk: The Analysis of Localisation Thresholds for Natural Sound Sources
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Objective Evaluation Techniques for Pairwise Panning-Based Stereo Upmix Algorithms for Spatial Audio

Department of Signal Processing and Communications, Helmut Schmidt University, Hamburg 22043, Germany
*
Author to whom correspondence should be addressed.
Appl. Sci. 2017, 7(4), 374; https://doi.org/10.3390/app7040374
Submission received: 16 January 2017 / Revised: 24 March 2017 / Accepted: 31 March 2017 / Published: 10 April 2017
(This article belongs to the Special Issue Spatial Audio)

Abstract

:
Techniques for generating multichannel audio from stereo audio signals are supposed to enhance and extend the listening experience of the listener. To assess the quality of such upmix algorithms, subjective evaluations have been carried out. In this paper, we propose an objective evaluation test for stereo-to-multichannel upmix algorithms. Based on defined objective criteria and special test signals, an objective comparative evaluation is enabled in order to obtain a quantifiable measure for the quality of stereo-to-multichannel upmix algorithms. Therefore, the basic functional principle of the evaluation test is demonstrated, and it is illustrated how possible results can be visualized. In addition, the proposed issues are introduced for the optimization of upmix algorithms and also for the clarification and illustration of the impacts and influences of different modes and parameters.

1. Introduction

While multichannel loudspeaker systems for home and car entertainment are becoming increasingly popular these days, the number of available multichannel audio recordings is still limited (note that multichannel audio recordings and mixed multichannel audio is meant subsequently). In contrast to movies on DVD or Blu-ray, the majority of audio recordings are only obtainable in the two-channel stereo format. In addition, the main content of digital radio and television and also the increasing significant streaming services for music and movies are only obtainable in the two-channel stereo format, too. So, it can be noted that there is a low availability of multichannel audio records. That is why a system is worthwhile, which extends an original stereo audio signal for playback over a multichannel loudspeaker system. As a result, the spatial quality and the listening experience can be enhanced compared with the pure stereo playback. For this reason, there is wide scholarly interest in novel stereo-to-multichannel upmix algorithms, e.g., [1,2,3,4,5] but there are many more recent publications on this topic. In order to determine the quality of such upmix algorithms, subjective listening tests were typically used [6,7,8]. Usher [6] presented specific design criteria for stereo-to-multichannel upmix algorithms to enhance spatial sound quality. He used formal listening tests for subjective evaluation of the design criteria according to [9], where three general sound quality issues for the evaluation of multichannel audio systems were defined. Choisel and Wickelmaier [7] used eight selected spatial attributes for sound quality evaluation in order to compare upmix algorithms. They derived a set of objectives measures from sound field analysis to predict auditory attributes. Barry and Kearney [8] used subjective listening tests for the assessment of source separation-based upmixing algorithms. In addition, they used objective testing to measure the errors which could theoretically occur in source separation algorithms.
Formal listening tests have been the only appreciable approach to assess the quality of stereo-to-multichannel upmix algorithms. More importantly, subjective quality assessments are always connected with expenditure and are both time-consuming and expensive. That is why an objective evaluation technique for stereo upmix algorithms for spatial audio is desirable.

2. Objective Evaluation Test

For the objective evaluation test, the following assumptions about stereo-to-multichannel upmix algorithms are made: stereo-to-multichannel upmix algorithms should enhance and extend the listening experience without adding artificial effects or contents and provide virtual sound sources true to original. The virtual sound sources in an original stereo configuration are placed between the two (front) loudspeakers. So, it is assumed that upmix algorithms are designed to have no virtual sound sources in the rear, only between the front loudspeakers. Therefore, the remaining amount of direct signal in the surround channels should be as low as possible. Furthermore, stereo-to-multichannel upmix algorithms should provide the listener with front channels that are louder than the surround channels under the condition that there is always an existing virtual sound source in the used stereo input signal. In addition, it is assumed that the effects of the correlation of the surround channels are perceived subjective, and that the surround channels should have a certain correlation to prevent uncomfortable perception. Finally, stereo-to-multichannel algorithms should create a high subjective perceived spatial quality with all loudspeakers in order to enhance the listening experience.
For the objective evaluation of stereo-to-multichannel upmix algorithms, the following tests were defined: 1. panning test, 2. direct signal test, 3. volume test, 4. phase test, 5. perception test. In every single test a special test signal is used as input signal for the tested upmix. The generated output signals are then analyzed and evaluated according to defined criteria (see Figure 1). Note that the evaluation test will measure how well the assumptions were met according to defined criteria.
The overall evaluation score s c o r e u p m i x of a tested upmix results from the weighted single-test evaluation scores s c o r e i , and is given by
s c o r e u p m i x = i g i · s c o r e i i g i
The higher a score, the better the test result. Zero is the worst, one the best evaluation score. Appropriate results should be visualized here based on two upmix algorithms available on the market with two modes for music (a) and movies (b) in each case. Hereinafter, they are denoted as upmix 1(a), 1(b), 2(a) and 2(b).

2.1. Panning Test

Criterion: The direction of the virtual sound source in the stereo-to-multichannel upmix should correspond to the direction of the virtual sound source in the initial stereo configuration. This is accompanied with the result that the spatial representation of sound events is preserved true to original.
The panning test (see Figure 2) is conducted in two versions. The evaluation score s c o r e P T results from the evaluation scores of the time- and frequency-independent panning test s c o r e P T 1 and the time- and frequency-dependent panning test s c o r e P T 2 , weighted with g P T 1 and g P T 2 , given by
s c o r e P T = g P T 1 · s c o r e P T 1 + g P T 2 · s c o r e P T 2 g P T 1 + g P T 2
Initially, at an interval of 1° virtual test sound sources φ i are defined with angles from −30° to 30° according to the reference loudspeaker arrangement [10]. With the tangent law as the modified stereophonic law of sines, the two panning coefficients a L , i and a R , i can be calculated from φ i [11]. A signal, weighted with the left panning coefficient a L , i , represents the left part of a stereo signal x L , i . A signal, weighted with the right panning coefficient a R , i , represents the right part of a stereo signal x R , i . For every angle φ i a stereo test signal is generated as input signal for the tested upmix. This is done by multiplying the resulting panning coefficients a L , i and a R , i with white Gaussian noise x w g n according to
x L , i ( n ) = a L , i · x w g n ( n ) x R , i ( n ) = a R , i · x w g n ( n )
With the output signals of the upmix algorithm for the three front channels x F L , i (front left), x C , i (center) and x F R , i (front right), two panning coefficients a ^ L , i and a ^ R , i are determined, and from these panning coefficients the direction of the virtual sound source φ ^ i is calculated. Note that two-to-five upmix algorithms are tested, but only the three front channels are used for the panning test. The difference Δ φ i = φ i φ ^ i , which is the deviation of the angle of the virtual sound source of the upmix from the defined test signal, serves as the basis for evaluation. The score of the time- and frequency-independent panning test s c o r e P T 1 is calculated from the mean of the normalized absolute deviation of all test cases with φ = ( φ 1 , , φ m )   W ( 1 i m ) and W = { n | 30 n 30 } given by
s c o r e P T 1 = 1 1 max ( | φ | ) · 1 m i = 1 m | Δ φ i |
The division by max ( | φ | ) is needed to normalize the mean of the deviations so that the evaluation score assumes values ranging from 0 to 1.
In the case of the time- and frequency-dependent panning test, the angle of the virtual test sound source is randomly generated in the range of −30° to 30°. This is done at any time n and frequency k in a time–frequency representation with the help of a short-time Fourier transform (STFT). So, the ability of the upmix to respond to fast changes of the virtual sound source should be tested. The score is calculated from the mean of the normalized absolute deviation across all N times and K frequencies with φ m a x as the maximum absolute value of all angles given by
s c o r e P T 2 = 1 1 φ m a x · 1 N · 1 K n = 1 N k = 1 K | Δ φ ( n , k ) |
The evaluation allows the comparison of the angle of the virtual sound source of the stereo input signal with the angle of the virtual sound source of the multichannel output signal (see Figure 3).
Upmix 1: The more the angle of the virtual sound source in the stereo configuration diverges from 0 ° in mode (a), the larger are the discrepancies in the multichannel configuration. As a consequence, the spatial extent of the initial stereo configuration is partly reduced significantly. At the same time, the majority of sound events is perceived from a small spot around the center. In mode (b) the direction of the virtual sound source in the multichannel configuration does not even tendentially comply with the direction of the virtual sound source in the stereo configuration. That is because of the aim of mode (b) to enhance speech intelligibility. So, only a small range around the center speaker ( φ i = 0 ° ) is emphasized and parts straight beyond this area are already located considerably further away.
Upmix 2: In mode (a) the angle of the virtual sound source in the multichannel configuration complies tendentially with the angle of the virtual sound source in the stereo configuration. The more φ i diverges from 0 ° , the larger are the discrepancies in the multichannel configuration until the angle φ ^ i converges fast towards the angle φ i . The spatial extent admittedly nearly remains, but the majority of the virtual sound sources is located closer to the center speaker. In mode (b) the spatial extent admittedly nearly remains, but within a certain area beyond the center ( φ i = 0 ° ) all sound events are solely located in one direction. To enhance speech intelligibility, sound events in the center are emphasized because parts straight beyond this area are located considerably further away.

2.2. Direct Signal Test

Criterion: The remaining amount of direct signal in the surround channels of the stereo-to-multichannel upmix could result in undesired virtual sound sources, which could interfere with the spatial representation of sound events true to original. Although it could lead to a higher subjective perceived spatial quality, the remaining amount of direct signal in the surround channels would be against the assumptions made for upmix algorithms, and should therefore be as low as possible.
Again, a special test signal is defined and used as input signal for the tested upmix algorithm. Different direct signals were taken from the database MedleyDB [12]. These audio recordings were then convolved with room impulse responses and mixed to a test signal (see Appendix A). The procedure of the direct signal test is shown in Figure 4.
The generated surround channels x S L (surround left) and x S R (surround right) of the upmix are analyzed by determining their remaining amount of direct signal S ^ A . This is compared with the known test signal S and serves as the basis for evaluation. The score of the direct signal test
s c o r e D T = 1 1 N · 1 K n = 1 N k = 1 K S ^ A , e n v ( n , k ) S e n v ( n , k )
is calculated from the mean of the quotient of the spectral envelopes [13] S ^ A , e n v and S e n v of the amounts of direct signals, across all N times and K frequencies, representing their relative deviation. To ensure a comparative evaluation, the summed power of the extracted surround channels must be equivalent to the summed power of the input signals. That is because signals before and after the upmixing process are considered. Only through using normalized surround signals is a comparative evaluation possible. This ensures, among others, that surround signals are considered correctly, which are identically equal to the input signals but reduced in power. That is because they would have the same relative amount of direct signal. The evaluation allows the comparison of the remaining surround channel direct signal with the known direct signal of the used test signal (see Figure 5).
Upmix 1: Mode (a) contains a reduced remaining direct signal in the surround channels. Mode (b) contains a remaining direct signal which is slightly lower or greater.
Upmix 2: The remaining direct signals are almost identical in modes (a) and (b), but slightly increased relative to the direct signal of the test signal. These proportionally increased remaining surround channel direct signals can occur because of input signal level adjustment or positive feedback of the upmix output signals. It should be noted that other upmix algorithms could also exhibit obviously reduced remaining surround channel direct signals.

2.3. Volume Test

Criterion: Power and loudness of the surround channels of the stereo-to-multichannel upmix should not be greater than the ones of the front channels. No unnatural or unexpected spatial sound should occur because the volume of the surround sound lateral or behind the listener is perceived louder than the volume of the sound events in front of the listener.
The volume test is therefore subdivided into the power test and the loudness test. The evaluation score of the volume test s c o r e L T results from the evaluation scores of the power test s c o r e L T 1 and the loudness test s c o r e L T 2 , weighted with g L T 1 and g L T 2 , and leads to
s c o r e L T = g L T 1 · s c o r e L T 1 + g L T 2 · s c o r e L T 2 g L T 1 + g L T 2
In each case, five-second-long extracts were taken from twelve popular pieces of music from various genres to create a sixty-second-long test signal (see Appendix B, Table A3).

2.3.1. Power Test

The procedure of the power test is shown in Figure 6.
The defined stereo test signal is used as input signal for the tested upmix. The power P F L , P C , P F R , P S L and P S R of the generated upmix output signals x F L , x C , x F R , x S L and x S R are considered. The maximum power of the three front signals P m a x is compared with the power of both surround signals P S L and P S R each. The ratios P L and P R serve as the basis for evaluation. The evaluation scores of the left and right surround signal, s c o r e L T 1 , L and s c o r e L T 1 , R , are calculated from the means of the quotients across all N L and N R times in which the power of the particular surround signals is greater than the power of the front channels, with P m a x ( n ) = m a x [ P F L ( n ) , P C ( n ) , P F R ( n ) ] :
s c o r e L T 1 , L = 1 1 N L n = 1 N L P L ( n )     P S L ( n ) P m a x ( n ) s c o r e L T 1 , R = 1 1 N R n = 1 N R P R ( n )     P S R ( n ) P m a x ( n )
The evaluation scores s c o r e L T 1 , L and s c o r e L T 1 , R represent the relative deviations of the considered power P S L respectively P S R from the maximum power of the three front signals P m a x . The evaluation score of the power test s c o r e L T 1 results from the evaluation scores of the power test for the left and right surround signal, s c o r e L T 1 , L and s c o r e L T 1 , R , weighted with g L T 1 , L and g L T 1 , R , and is given by
s c o r e L T 1 = g L T 1 , L · s c o r e L T 1 , L + g L T 1 , R · s c o r e L T 1 , R g L T 1 , L + g L T 1 , R
The evaluation allows the comparison of front with surround channel power (see Figure 7). For reasons of clarity only the left surround channel power is used in the following figures.
Upmix 1: In mode (a), the power of the left surround channel is basically lower and in some areas partly as high as the power of the front channel with the greatest power. In mode (b), the power of the left surround channel is mostly lower and in some areas partly higher than the power of the front channel with the greatest power. While in mode (a), the power of each front channel is more or less relatively similar, they are mostly considerably different in mode (b). The strong emphasis on the center channel can especially be recognized.
Upmix 2: In mode (a), the power of the left surround channel is basically lower and in some areas partly as high as the power of the front channel with the greatest power. In mode (b), the power of the left surround channel is lower and in some areas partly as high as or slightly higher than the power of the front channel with the greatest power. While in mode (a), the power of each front channel is more or less relatively similar, they are mostly considerably different in mode (b). The strong emphasis on the center channel can especially be recognized.

2.3.2. Loudness Test

The procedure of the loudness test is shown in Figure 8.
The defined stereo test signal is used as input signal for the tested upmix. The generated upmix output signals x F L , x C , x F R , x S L and x S R are considered. The loudness L K , f r o n t of the three front channels x F L , x C and x F R is compared with the loudness L K , r e a r of both surround channels x S L and x S R . This serves as the basis for the evaluation score of the loudness test s c o r e L T 2 . The determination of the loudness is done blockwise according to [14], but separately for the loudness of the front channels L K j , f r o n t and the loudness of the surround channels L K j , r e a r . The evaluation score is calculated from the mean of the absolute deviations Δ L K j = L K j , r e a r L K j , f r o n t across all J blocks in which the loudness of the surround channels is greater than the loudness of the front channels, and is given by
s c o r e L T 2 = 1 1 J   j J Δ L K j     L K j , r e a r L K j , f r o n t
The evaluation allows the comparison of the loudness of the front channels with the loudness of the surround channels (see Figure 9).
Upmix 1: In mode (a), the loudness of the surround channels is basically lower and in some areas partly similar or slightly greater than the loudness of the front channels. In mode (b), the loudness of the surround channels is basically lower and in some areas partly higher than the loudness of the front channels.
Upmix 2: In mode (a) as well as in mode (b), the loudness of the surround channels is basically lower and in some areas partly similar or slightly greater than the loudness of the front channels.

2.4. Phase Test

Criterion: The surround channels of the stereo-to-multichannel upmix should have a certain correlation to prevent uncomfortable perception. If the surround channels would be completely correlated, a mono sound source would be created, which could be perceived as uncomfortable. If the surround channels would be completely decorrelated, two independent sound sources would be created, which could be perceived as uncomfortable, too [15,16,17,18].
The procedure of the phase test is shown in Figure 10. The test signal is identically equal to the test signal of the volume test and is used as input signal for the tested upmix.
The correlation degree Υ S L S R results from the normalized cross-correlation of the generated upmix output signals x S L and x S R for τ = 0 , and leads with
Υ ε ( n ) = { | Υ S L S R ( n ) | ε o 1 ε o ,   | Υ S L S R ( n ) | > ε o ε u | Υ S L S R ( n ) | ,   | Υ S L S R ( n ) | < ε u 0 ,   ε u | Υ S L S R ( n ) | ε o
to the evaluation score of the phase test s c o r e P h T according to
s c o r e P h T = 1 1 N n = 1 N Υ ε ( n )
With the requirement that the surround channels x S L and x S R should not be either completely correlated or completely decorrelated, two evaluation limits, ε o = 0.5 and ε u = 0.2 , were defined within which a certain correlation is supposed to be optimal. The tendency for complete correlation and thus the creation of a mono sound source is higher weighted in the evaluation score than the tendency for complete decorrelation. The evaluation allows the comparison of the correlation degrees of the surround channels (see Figure 11).
Upmix 1: In mode (a), the correlation degree of the surround channels is basically negative, in mode (b), basically positive. It is notable that the surround signals of the one mode are a phase-inverted version of the surround signals of the other mode.
Upmix 2: In both modes, the correlation degree of the surround channels is basically approximately one. So, there is the danger that the correlated surround signals are decomposed into a mono signal.

2.5. Perception Test

Criterion: The stereo-to-multichannel upmix should generate a high subjectively perceived spatial quality. This is accompanied with the result that the listening experience is improved compared to the initial stereo configuration, and that the listener feels projected in the middle of the sound events.
The procedure of the perception test is shown in Figure 12. The test signal is identically equal to the test signal of the volume and phase test and is used as input signal for the tested upmix.
The interaural cross-correlation coefficient (IACC) describes the subjectively perceived spatial quality of sound events, and is a measure for apparent source width (ASW). The lateral energy fraction (LF) describes the impression of spatial quality, and is also a measure for listener envelopment (LEV) [7,19,20]. For the determination of IACC, the generated upmix output signals x F L , x C , x F R , x S L and x S R are used to create a simulated sound field on the basis of head-related impulse responses (HRIR). The binaural signals:
p L ( n ) = i x i ( n ) h i , L ( n ) p R ( n ) = i x i ( n ) h i , R ( n )
result from the summed generated upmix output signals x i across all channels i of the multichannel configuration convolved with the particular head-related impulse responses h i , L and h i , R for the left and right ear (with i I and I = { F L ,   C ,   F R ,   S L ,   S R } ).
IACC results from the normalized cross-correlation of the half-wave rectified and with a third-order Butterworth filter ( f c = 1   kHz ) low-pass filtered binaural signals p L and p R . The prefiltering ensures that the results correspond better to the subjectively perceived spatial quality [21,22,23].
For the determination of LF, the generated upmix output signals x F L , x C , x F R , x S L and x S R are used to create another simulated sound field. The signal p 0 , which is recorded from a virtual omnidirectional microphone, results from the summed generated upmix output signals x i across all channels i of the multichannel configuration, and is given by
p 0 ( n ) = i x i ( n )
The signal p 8 , which is recorded by a virtual bidirectional microphone, results from the summed generated upmix output signals x i across all channels i of the multichannel configuration weighted with the respective loudspeaker directions φ i , and can be written as
p 8 ( n ) = i x i ( n ) · c o s ( φ i )
LF results, with the signals p 8 and p 0 , from the ratio of acoustic waves, which are arriving at the listening position laterally and from all directions [7,23], given by
L F = n = 1 N p 8 2 ( n ) n = 1 N p 0 2 ( n )
Due to using a simulated sound field, the differentiation between early-and late-arriving signal components is omitted [7].
The evaluation score of the perception test based on IACC results in
s c o r e W T 1 = 1 I A C C
and is a direct measure for ASW. The subjectively perceived spatial quality is the higher, the lower IACC is. The evaluation score s c o r e W T 2 of the perception test based on LF results in
s c o r e W T 2 = L F
and is a direct measure for LEV. The impression of spatial quality is the higher, the higher LF is. The evaluation score of the perception test s c o r e W T results from the evaluation scores of the perception test based on IACC and LF, s c o r e W T 1 and s c o r e W T 2 , weighted with g W T 1 and g W T 2 , given by
s c o r e W T = g W T 1 · s c o r e W T 1 + g W T 2 · s c o r e W T 2 g W T 1 + g W T 2
Use of simulated sound fields within the scope of the perception test ensures simplicity because of independence from the properties of room, speakers, microphones, etc., which had to be considered for the determination of IACC and LF based on costly recordings. The evaluation allows the comparison of IACC and LF (see Figure 13).
Upmix 1: In mode (a), IACC assumes middle to high values, LF assumes high values (see Figure 13). According to that, middle to low subjectively perceived spatial quality and a high impression of spatial quality occurs. In mode (b), IACC and LF assume middle to high values. According to that, middle to low subjectively perceived spatial quality and middle to high impression of spatial quality occurs.
Upmix 2: In both modes, IACC assumes high and LF middle to high values (see Figure 14). According to that, low subjectively perceived spatial quality and middle to high impression of spatial quality occurs.

3. Results

Table A4 (Appendix C) summarizes the single scores of all evaluation tests for the exemplarily tested stereo-to-multichannel upmix algorithms, the used weighting factors and the resulting overall evaluation score. The higher a score, the better the test result of the tested stereo-to-multichannel upmix according to the defined criteria. Zero is the worst, one the best evaluation score. With the help of weighting factors, significance of single tests can be adjusted (see Appendix D). Figure 14 illustrates the evaluation scores according to Table A4. All in all, upmix 1(a) has the best overall evaluation score by far, upmix 1(b) the second best. Upmix 2(b) has the worst overall evaluation score, upmix 2(a) the second worst. Note that two commercial upmix algorithms in two different modes were used to demonstrate the functional principle of the proposed evaluation test and to illustrate how possible results can be visualized. The aim of this paper was not to compare existing upmix algorithms but to introduce an objective evaluation test to gain the possibility of objective comparison. So, an overall evaluation of an upmix algorithm with the proposed evaluation test is appropriate in comparison with other upmix algorithms as references. Therefore, Figure 14 provides an appropriate graphical overview for the comparative evaluation of different upmix algorithms. Note that corresponding results of the single evaluation tests were presented in the end of each section.
For the proposed evaluation test several assumptions about stereo-to-multichannel upmix algorithms were made. Since upmix algorithms are also based on assumptions, the evaluation test will measure how well the assumptions made here were met. Furthermore, a self-contained evaluation of a single upmix algorithm should focus on panning test, direct signal test and volume test. That is because the effects of the correlation of the surround channels (phase test) are perceived subjective. In addition, the impacts of lateral energy fraction and interaural cross-correlation (perception test) on perceived spatiality are subjective, too.

4. Conclusions

In this paper, we proposed an objective evaluation for stereo-to-multichannel upmix algorithms based on defined objective criteria, special test signals and several single evaluation tests. Two upmix algorithms available on the market were used to demonstrate the single tests exemplarily. The panning test checks whether the direction of the virtual sound source in the stereo-to-multichannel upmix corresponds to the direction of the virtual sound source in the initial stereo configuration. The direct signal test checks whether the remaining direct signal in the surround channels is as low as possible. The volume test checks whether the power and the loudness of the surround channels is not greater than these of the front channels. The phase test checks whether the surround channels of the stereo-to-multichannel upmix are not either completely correlated or completely decorrelated, but have a certain correlation. And the perception test checks whether the stereo-to-multichannel upmix generates a high subjectively perceived spatial quality.
The introduced objective evaluation test enables an objective comparative evaluation, which can now provide a measurable quantity for the quality of stereo-to-multichannel upmix algorithms. In addition, the objective evaluation test could be used for the optimization of upmix algorithms and also for the clarification and illustration of the impacts and influences of different modes and parameters. The proposed objective evaluation test is assumed as an appropriate alternative or supplement for time-consuming and expensive subjective listening tests.
Nevertheless, a comparison of the proposed objective evaluation test with subjective test results will be a focus of future work as part of appropriate validation.

Acknowledgments

This research was funded by Helmut Schmidt University, Hamburg, Germany.

Author Contributions

Martin Mieth is the first author, the developer of this research and wrote the paper. Udo Zölzer is the corresponding author, managed the overall research, supervised the complete work and edited the manuscript.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

The direct signal used for the direct signal test results (according to Table A1) in
s ( n ) = m s m ( n )
with m M and M = { v o c a l s ,   b a s s ,   d r u m s ,   r a t t l e ,   g u i t a r } . The two-channel test signal results in
x L ( n ) = m s m ( n ) h m , L x R ( n ) = m s m ( n ) h m , R
The room impulse responses h m , L und h m , R were taken from the Aachen Impulse Response (AIR) Database [24] with the following parameters:
  • type of room impulse response: Binaural; recorded with a dummy head.
  • type of room: Stairway.
  • distance of sound event to listening position: 3 m.
  • angle of sound events: see Table A2.
Table A1. Used direct signals from MedleyDB.
Table A1. Used direct signals from MedleyDB.
Sound EventFile from DatabaseVariable
vocalsAClassicEducation_NightOwl_RAW_08_01.wav s v o c a l s
bassAClassicEducation_NightOwl_RAW_01_02.wav s b a s s
drumsAClassicEducation_NightOwl_RAW_02_01.wav s d r u m s
rattleAClassicEducation_NightOwl_RAW_11_01.wav s r a t t l e
guitarAClassicEducation_NightOwl_RAW_05_01.wav s g u i t a r
Table A2. Angle of sound events.
Table A2. Angle of sound events.
Sound EventAngle of Sound Events 1
vocals90°
bass
drums135°
rattle45°
guitar180°
1 Different directions were provided for the single sound events. Here, it is: 0° left, 90° front, 180° right.

Appendix B

Table A3. Used pieces of music.
Table A3. Used pieces of music.
GenreInterpreterTitleTime of TitleTime of Test Signal
popThe Human LeagueDon‘t you want me1:18–1:230:00–0:05
popMichael JacksonThriller1:27–1:320:05–0:10
popLittle TalksOf Monsters and Men0:04–0:090:10–0:15
rockMetallicaEnter Sandmann0:57–1:020:15–0:20
rockMotörheadAce of Spades0:13–0:180:20–0:25
rockBlack SabbathParanoid0:12–0:170:25–0:30
danceThe ProdigyOmen0:09–0:140:30–0:35
danceThe ChainsmokersSelfie0:38–0:430:35–0:40
classicAntonio VivaldiThe Four Seasons1:10–1:150:40–0:45
classicCarl OrffO Fortuna0:08–0:130:45–0:50
jazzLouis ArmstrongWhat a wonderful world1:29–1:340:50–0:55
jazzQuincy JonesPink Panther Theme1:48–1:530:55–1:00

Appendix C

Table A4. Evaluation scores of the upmix algorithms—tabular overview.
Table A4. Evaluation scores of the upmix algorithms—tabular overview.
Test ScoreUpmix 1(a)Upmix 1(b)Upmix 2(a)Upmix (2b)Weighting
s c o r e P T 1 0.78760.81650.80450.8155 g P T 1 = 2
s c o r e P T 2 0.63930.62960.63730.6510 g P T 2 = 1
s c o r e P T 0.73820.75420.74880.7607 g P T = 3
s c o r e D T 0.2727000 g D T = 2
s c o r e L T 1 , L 0.84840.67940.87960.7751 g L T 1 , L = 1
s c o r e L T 1 , R 0.82410.66860.87420.7581 g L T 1 , R = 1
s c o r e L T 1 0.83620.67400.87690.7666 g L T 1 = 1
s c o r e L T 2 0.186600.43870.1019 g L T 2 = 1
s c o r e L T 0.51140.33700.65780.4342 g L T = 2
s c o r e P h T 0.98510.97290.07370.0468 g P h T = 1
s c o r e W T 1 0.28940.22830.20440.1867 g W T 1 = 1
s c o r e W T 2 0.86260.67880.71410.7178 g W T 2 = 1
s c o r e W T 0.57600.45360.45930.4522 g W T = 1
s c o r e U p m i x 0.59380.48480.45500.4055

Appendix D

The proposed objective evaluation test uses weighting factors to determine an overall evaluation score from the weighted single-test evaluation scores. With the help of these weighting factors, significance of single tests can be adjusted.
The panning test is conducted in two versions. The aim of the time- and frequency-independent panning test is to test the ability of an upmix algorithm to reproduce the virtual sound sources true to original. Therefore, time- and frequency-independent virtual test sound sources are defined. The aim of the time- and frequency-dependent panning test is to test the ability of the upmix algorithm to respond to fast changes of the virtual sound source. Therefore, time- and frequency-dependent virtual test sound sources are generated independently. Since the focus of the panning test is whether the direction of the virtual sound source in the stereo-to-multichannel upmix corresponds to the direction of the virtual sound source in the initial stereo configuration, the time- and frequency-independent panning test is higher weighted than the time- and frequency-dependent panning test (the time- and frequency-independent panning test counts twice). The evaluation score of the panning test results from the double-weighted time- and frequency-independent panning test ( g P T 1 = 2 ) and the single-weighted time- and frequency-dependent panning test ( g P T 2 = 1 ). Due to the assumed high overall importance of the panning test in comparison with the other single tests, the panning test is triple-weighted ( g P T = 3 ). That is because a stereo-to-multichannel upmix should enhance and extend the listening experience without adding artificial effects or contents, and provide virtual sound sources true to original.
The phase test and the perception test are single-weighted ( g P h T = 1 ; g W T = 1 ) because of the assumed low overall importance in comparison to the other single evaluation tests. That is because the effects of the correlation of the surround channels are perceived subjective. In addition, the impacts of lateral energy fraction and interaural cross-correlation on perceived spatiality are subjective, too. The direct test and the volume test are double-weighted ( g D T = 2 ; g L T = 2 ) because of the assumed middle overall importance in comparison to the other single evaluation tests. Thus, they are higher weighted than the phase test and the perception test, but lower weighted than the panning test.
Note that the weighting factors used are not obligatory. They can be used in order to adjust the significance of the single evaluation tests as described above. In addition, they can be used to compare several upmix algorithms with a focus on a special issue.

References

  1. Avendano, C.; Jot, J.-M. Frequency Domain Techniques for Stereo to Multichannel Upmix. In Proceedings of the 22nd International Conference on Virtual, Synthetic and Entertainment Audio (AES), Espoo, Finland, 15–17 June 2002. [Google Scholar]
  2. Faller, C. Multiple-Loudspeaker Playback of Stereo-Signals. Jt. Audio Eng. Soc. 2006, 54, 1051–1064. [Google Scholar]
  3. Goodwin, M.M.; Jot, J.-M. Primary-Ambient Signal Decomposition and Vector-Based Localization for Spatial Audio Coding and Enhancement. In Proceedings of the 2007 IEEE International Conference on Acoustics, Speech and Signal Processing, Honolulu, HI, USA, 16–20 April 2007; pp. 9–12. [Google Scholar]
  4. Vickers, E. Frequency-Domain Two-to-Three-Channel Upmix for Center Channel Derivation and Speech Enhancement. In Proceedings of the AES 127th Convention, New York, NY, USA, 9–12 October 2009. [Google Scholar]
  5. Jeon, S.-W.; Park, Y.-C.; Lee, S.-P.; Youn, D.-H.H. Robust Representation of Spatial Sound in Sterei-to-Multichannel Upmix. In Proceedings of the AES 128th Convention, London, UK, 22–25 May 2010. [Google Scholar]
  6. Usher, J.S. Subjective Evaluation and Electroacoustic Theoretical Validation of a New Approach to Audio Upmixing. Ph.D. Thesis, McGill University, Montreal, QC, Canada, September 2006. [Google Scholar]
  7. Choisel, S.; Wickelmaier, F. Relating Auditory Attributes of Multichannel Sound to Preference and to Physical Parameters. In Proceedings of the AES 120th Convention, Paris, France, 20–23 May 2006. [Google Scholar]
  8. Barry, D.; Kearney, G. Localization Quality Assessment in Source Separation-Based Upmixing Algorithms. In Proceedings of the AES 35th International Conference, London, UK, 11–13 February 2009. [Google Scholar]
  9. ITU-R BS 1116-0. Methods for the Subjective Assessment of Small Impairments in Audio Systems Including Multichannel Sound Systems; Recommendation ITU-R BS 1116-0; International Telecommunication Union: Geneva, Switzerland, July 1994. [Google Scholar]
  10. ITU-R BS.775-3. Multichannel Stereophonic Sound System with and without Accompanying Picture; Recommendation ITU-R BS.775-3; International Telecommunication Union: Geneva, Switzerland, August 2012. [Google Scholar]
  11. Pulkki, V. Virtual sound source positioning using vector base amplitude panning. Jt. Audio Eng. Soc. 1997, 45, 457. [Google Scholar]
  12. Bittner, R.; Salamon, J.; Tierney, M.; Mauch, M.; Cannam, C.; Bello, J.P. MedleyDB: A Multitrack Dataset for Annotation-Intensive MIR Research. Proceeding of the 15th International Society for Music Information Retrieval Conference, Taipei, Taiwan, 27–31 October 2014. [Google Scholar]
  13. Röbel, A.; Rodet, X. Efficient Spectral Envelope Estimation and Its application to Pitch Shifting and Envelope Preservation. In Proceedings of the 8th International Conference on Digital Audio Effects (DAFx05), Madrid, Spain, 20–22 September 2005; pp. 30–35. [Google Scholar]
  14. ITU-R BS.1770-3. Algorithms to Measure Audio Programme Loudness and True-Peak Audio Level; Recommendation ITU-R BS.1770-3; International Telecommunication Union: Geneva, Switzerland, August 2012. [Google Scholar]
  15. Riekehof-Böhmer, H.; Wittek, H. Prediction of perceived width of stereo microphone setups. In Proceedings of the AES 130th Convention, London, UK, 13–16 May 2011. [Google Scholar]
  16. Theile, G. Multichannel Natural music Recording Based on Psychoacoustic Principles. In Proceedings of the AES 19th International Conference, Schloss Elmau, Germany, 21–24 June 2001. [Google Scholar]
  17. Damaske, P. Subjektive Untersuchung von Schallfeldern. Acta Acust. 1967, 19, 68. [Google Scholar]
  18. Usher, J.S. Design Criteria for High Quality Upmixers. In Proceedings of the AES 28th International Conference, Piteå, Sweden, 30 June–2 July 2006. [Google Scholar]
  19. Hirst, J.M. Spatial Impression in Multichannel Surround Sound Systems. Ph.D. Thesis, University of Salford, Salford, UK, 2006. [Google Scholar]
  20. Bradley, J.S.; Soulodre, G.A. The influence of late arriving energy on spatial impression. J. Acoust. Soc. Am. 1995, 97, 2590–2597. [Google Scholar] [CrossRef]
  21. Mason, R.; Brooks, T.; Rumsey, F. Frequency dependency of the relationship between perceived auditory source width and the interaural cross-correlation coefficient for time-invariant stimuli. J. Acoust. Soc. Am. 2005, 117, 1337–1350. [Google Scholar] [CrossRef] [PubMed]
  22. Neher, T.; Brookes, T.; Mason, R. Musically representative test signals for interaural cross-correlation coefficient measurement. Acta Acoust. 2006, 92, 787–796. [Google Scholar]
  23. Bradley, J.S. Comparison of concert hall measurements of spatial impression. J. Acoust. Soc. Am. 1994, 96, 3525–3535. [Google Scholar] [CrossRef]
  24. Jeub, M.; Schäfer, M.; Vary, P. A Binaural Room Impulse Response Database for the Evaluation of Dereverberation Algorithms. In Proceedings of the 2009 16th International Conference on Digital Signal Processing (DSP), Santorini, Greece, 5–7 July 2009. [Google Scholar]
Figure 1. Schematic procedure of the evaluation tests.
Figure 1. Schematic procedure of the evaluation tests.
Applsci 07 00374 g001
Figure 2. Block diagram: Time- and frequency-independent panning test.
Figure 2. Block diagram: Time- and frequency-independent panning test.
Applsci 07 00374 g002
Figure 3. Panning test—angle of the virtual test sound sources compared with the determined virtual sound sources of the upmix algorithms: (a) algorithms upmix 1(a) and 1(b); (b) algorithms upmix 2(a) and 2(b).
Figure 3. Panning test—angle of the virtual test sound sources compared with the determined virtual sound sources of the upmix algorithms: (a) algorithms upmix 1(a) and 1(b); (b) algorithms upmix 2(a) and 2(b).
Applsci 07 00374 g003
Figure 4. Block diagram: Direct signal test. Note that this is a simplified illustration of room impulse responses as a summary. See Appendix A for details of used room impulse responses and the calculation of x L and x R . STFT: short-time Fourier transform.
Figure 4. Block diagram: Direct signal test. Note that this is a simplified illustration of room impulse responses as a summary. See Appendix A for details of used room impulse responses and the calculation of x L and x R . STFT: short-time Fourier transform.
Applsci 07 00374 g004
Figure 5. Direct signal test—remaining surround channel direct signal: (a) Algorithms upmix 1(a) and 1(b); (b) algorithms upmix 2(a) and 2(b). Note that the index n in the plots represents discrete values of time. | S ( n ) | represents the magnitude of the respective complex direct signal averaged over all frequencies.
Figure 5. Direct signal test—remaining surround channel direct signal: (a) Algorithms upmix 1(a) and 1(b); (b) algorithms upmix 2(a) and 2(b). Note that the index n in the plots represents discrete values of time. | S ( n ) | represents the magnitude of the respective complex direct signal averaged over all frequencies.
Applsci 07 00374 g005
Figure 6. Block diagram: Power test.
Figure 6. Block diagram: Power test.
Applsci 07 00374 g006
Figure 7. Power test: (a) Upmix 1(a); (b) upmix 1(b); (c) upmix 2(a); (d) upmix 2(b).
Figure 7. Power test: (a) Upmix 1(a); (b) upmix 1(b); (c) upmix 2(a); (d) upmix 2(b).
Applsci 07 00374 g007
Figure 8. Block diagram: Loudness test.
Figure 8. Block diagram: Loudness test.
Applsci 07 00374 g008
Figure 9. Loudness test: (a) Upmix 1(a); (b) upmix 1(b); (c) upmix 2(a); (d) upmix 2(b).
Figure 9. Loudness test: (a) Upmix 1(a); (b) upmix 1(b); (c) upmix 2(a); (d) upmix 2(b).
Applsci 07 00374 g009
Figure 10. Block diagram: Phase test. Correlation degree is calculated in the block “KOR”.
Figure 10. Block diagram: Phase test. Correlation degree is calculated in the block “KOR”.
Applsci 07 00374 g010
Figure 11. Correlation degree of surround channels: (a) Upmix 1(a) and 1(b); (b) upmix 2(a) and 2(b).
Figure 11. Correlation degree of surround channels: (a) Upmix 1(a) and 1(b); (b) upmix 2(a) and 2(b).
Applsci 07 00374 g011
Figure 12. Block diagram: Perception test. IACC: interaural cross-correlation coefficient; LF; lateral energy fraction; LP: low-pass.
Figure 12. Block diagram: Perception test. IACC: interaural cross-correlation coefficient; LF; lateral energy fraction; LP: low-pass.
Applsci 07 00374 g012
Figure 13. Perception test: (a) IACC for upmix 1(a) and 1(b); (b) IACC for upmix 2(a) and 2(b); (c) LF for upmix 1(a) and 1(b) IACC; (d) LF for upmix 2(a) and 2(b).
Figure 13. Perception test: (a) IACC for upmix 1(a) and 1(b); (b) IACC for upmix 2(a) and 2(b); (c) LF for upmix 1(a) and 1(b) IACC; (d) LF for upmix 2(a) and 2(b).
Applsci 07 00374 g013
Figure 14. Evaluation scores of the upmix algorithms—graphical overview.
Figure 14. Evaluation scores of the upmix algorithms—graphical overview.
Applsci 07 00374 g014

Share and Cite

MDPI and ACS Style

Mieth, M.; Zölzer, U. Objective Evaluation Techniques for Pairwise Panning-Based Stereo Upmix Algorithms for Spatial Audio. Appl. Sci. 2017, 7, 374. https://doi.org/10.3390/app7040374

AMA Style

Mieth M, Zölzer U. Objective Evaluation Techniques for Pairwise Panning-Based Stereo Upmix Algorithms for Spatial Audio. Applied Sciences. 2017; 7(4):374. https://doi.org/10.3390/app7040374

Chicago/Turabian Style

Mieth, Martin, and Udo Zölzer. 2017. "Objective Evaluation Techniques for Pairwise Panning-Based Stereo Upmix Algorithms for Spatial Audio" Applied Sciences 7, no. 4: 374. https://doi.org/10.3390/app7040374

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop