A Method Based on Dual Cross-Modal Attention and Parameter Sharing for Polyphonic Sound Event Localization and Detection

Abstract

Sound event localization and detection (SELD) is a joint task that unifies sound event detection (SED) and direction-of-arrival estimation (DOAE). The task has become such a popular topic that it was introduced into the challenge on Detection and Classification of Acoustic Scenes and Events (DCASE) Task3 in 2019. In this paper, we propose a method based on dual cross-modal attention (DCMA) and parameter sharing to simultaneously detect and localize sound events. In particular, the DCMA-based decoder commonly used for multiple predictions efficiently learns the associations between SED and DOAE features by exchanging SED and DOAE information in the process of attention, in addition to the encoder with parameter sharing. Furthermore, acoustic features that have not been usually used in the SELD task are additionally adopted to improve the performance, and data augmentation techniques of the mixup to simulate polyphonic events and channel rotation for spatial augmentation are conducted for this task. Experimental results demonstrate that our efficient model using one common decoder block based on the DCMA to predict multiple events in the track-wise output format is effective for the SELD task with up to three overlapping events.

1. Introduction

Sound event localization and detection (SELD) is an audio processing task that unifies the tasks of sound event detection (SED) and its source localization by jointly recognizing target classes of sound events including temporal information of the sound activations and estimating their directions of arrival (DOA) [1,2,3,4]. For example, it can be used in indoor or outdoor situations, such as a baby crying, a window broken, a car horn on the road, and a disaster relief call. In these situations, it is reasonable to combine detection and localization by estimating the temporal and spatial locations of the target events since the sound of an event is transmitted to microphones from the corresponding source in a specific direction. Therefore, SELD has various applications including acoustic monitoring [5], surveillance [6], autonomous driving, and robotics [7], and the task has become such a popular topic that it was introduced into the challenge on Detection and Classification of Acoustic Scenes and Events (DCASE) Task3 in 2019 [8]. Since sound sources should be localized, SELD typically requires multi-channel audio data acquired by a microphone array.

In DCASE2021, the TAU-NIGENS Spatial Sound Events (TNSSE) 2021 dataset is provided in two four-channel spatial audio formats, the raw signals of a tetrahedral microphone array (MIC) and first-order Ambisonics (FOA), which included emulation of recordings of static or moving sound events by spatial room impulse responses (SRIRs) captured in various rooms [3]. The isolated sound event recordings were obtained from the NIGENS general sound events database [9], and the spatialization of all sound events was based on filtering through spatial room impulse responses. Localized interfering events outside of the target classes and spatial ambient noise captured in these rooms were added to the recordings [3]. In this paper, we only used the FOA (four-channels, three-dimensional recordings) data format. The spatial responses of the four FOA channels to a source incident from DOA given by azimuth angle $\varphi$ and elevation angle $\theta$ are as follows [10]:

$$\begin{array}{ccc}\hfill H_1(\varphi ,\theta ,f)& =& 1,\hfill \end{array}$$

(1)

$$\begin{array}{ccc}\hfill H_2(\varphi ,\theta ,f)& =& cos\left(\varphi \right)cos\left(\theta \right),\hfill \end{array}$$

(2)

$$\begin{array}{ccc}\hfill H_3(\varphi ,\theta ,f)& =& sin\left(\varphi \right)cos\left(\theta \right),\hfill \end{array}$$

(3)

$$\begin{array}{ccc}\hfill H_4(\varphi ,\theta ,f)& =& sin\left(\theta \right).\hfill \end{array}$$

(4)

The FOA format is obtained by converting the 32-channel microphone array signals by means of encoding filters based on anechoic measurements of the Eigenmike array response.

The task contains 12 different SED target classes of alarm, crying baby, crash, barking dog, female scream, female speech, footsteps, knocking on the door, male scream, male speech, ringing phone, and piano, and directional interferers of running engine and burning fire, and an additional general class with disparate sounds not belonging to any of the other classes that have their own temporal activities from a static or moving source. Up to three overlapping target events may occur, and these events may belong to the same class. The dataset consists of 800 one-minute spatial recordings that are split into the development set of 600 recordings and the evaluation set of 200 recordings. The development set is split into six disjoint sets of 100 recordings each. In total, 500 data (fold 1, 2, 3, 4, 5) were used for model training, and the other 100 (fold 6) were used as the test set to permit evaluation across different acoustic conditions. Note that in this challenge, the use of external data is not allowed, and a pre-trained network cannot be used to develop a model. The dataset can be augmented any number of times under the condition that no external data are used.

1.1. Existing Methods

Although SED and sound source localization (SSL) have been independently studied for a long time, research on SELD that performs two tasks simultaneously has been actively conducted only recently, and it was first introduced to a DCASE Challenge task in 2019. Every year, the data handled by this task are updated with difficulty, and the problem to be solved is complicated. In 2020, some sound sources sometimes move. In 2021, up to three overlapping events will occur, and there may be directional noise interference rather than target events. Since not all directional sounds are target events, it is important to match the class and direction of a sound event that occurs. During and after the competition every year, various modeling, features extraction, data augmentation, or other methodologies were introduced by many participants and researchers.

In 2019, Cao et al. trained the SED and DOA estimation (DOAE) models separately, and they obtained the DOAE output only when the SED output predicted an event [11]. In 2020, they presented the Event Independent Network (EIN) that outputs track-wise predictions, which enables predicting events of the same class that occur at the same time. When extracting the features of SED and DOAE through the convolutional block, related features between the two tasks were well extracted by applying parameter sharing between the two embeddings [12]. After the challenge, an improved version of the EIN (EINV2) was released, which utilizes multi-head self-attention instead of bidirectional gated recurrent units (biGRUs) after the convolutional layers [13].

In 2020, Shimada et al. expressed the labels of SED and SSL as one using the activity-coupled Cartesian DOA vector (ACCDOA) representation [14]. Thus, they did not divide the task into two different ones and considered it as a single system that requires just one loss calculation to optimize their model. They proposed the network called RD3Net, which uses a dense block with skip connections and dilated convolutions. The ACCDOA vectors are generated as model results, and training is conducted in a direction that minimizes the distance between the coordinates indicated by this vector and the target coordinates. Therefore, they treated it as a multi-output regression problem rather than a classification problem and used a mean square error (MSE) loss. In 2021, a model was designed in a way that ensembles the ACCDOA-based system and EINV2-based system, and it achieved better performance than all the other methods submitted to the DCASE2021 Task3 [15].

Adavanne et al. used the SELDNet to jointly estimate polyphonic events of various classes and their source directions [1]. The multi-channel phase and magnitude spectrograms were used as the feature. Multiple layers of two-dimensional (2D) convolutional neural network (CNN) and GRU were followed by two branches of fully connected (FC) layers in parallel for SED and DOAE predictions. The SED and DOAE were performed as a multi-label classification task and a multi-output regression task, respectively. In particular, the modified version of this network was used as the reference system for Task3 in DCASE2019, DCASE2020, and DCASE2021 [3,10,16].

Nguyen et al. proposed a feature named spatial cue-augmented log-spectrogram (SALSA) which stacks multi-channel log-spectrograms with estimated direct-to-reverberant ratio (DRR) and the normalized principal eigenvectors of the spatial covariance matrix at each time-frequency bin on the spectrograms [2,17]. The normalized eigenvector for FOA corresponds to inter-channel intensity difference (IID) for the FOA format, and this feature outperformed the method based on intensity vector (IV). The ResNet22 network was fixed for the encoder, whereas three different networks were used for the decoder: long short-term memory (LSTM), GRU, and multi-head self-attention.

In 2020, Park et al. designed three kinds of loss functions to address the issue of data imbalance problems [18]. First, a temporal masking loss was used to mask the silence interval to give more weight to the interval when target events occur. Second, the soft loss was used to overcome the problem of data imbalance caused by the different number of target events belonging to each class. Third, a sinusoidal loss was used to deal with periodic label values for DOA, since angles in azimuth and elevation are periodic. In 2021, Park et al. used a self-attention-based model called many-to-many audio spectrogram transformer (M2M-AST) to achieve improved performance in the SELD [19].

1.2. Our Contributions

In this paper, we focus on the method of enhancing the information necessary for each other between the SED and DOAE features, since SELD is a task predicting the temporal and spatial locations of a target event as two different aspects of one signal. Our model is based on the encoder–decoder structure. For the encoder, we adopt the embedding scheme of the EINV2 using soft parameter sharing, while the decoder part of the transformer [20] is modified to predict either sound events or source directions. Different from the conventional transformer that analyzes sequence data by feeding back the previous output as the current decoder input, embedded SED or DOAE features are directly used as the input to analyze the current sound in our decoder.

In particular, we apply dual cross-modal attention (DCMA) in the decoder part, which is the first trial for multi-task learning including the SELD task as far as we know, although DCMA has been used in multi-modal tasks such as audio-visual speech recognition and audio-text emotion detection [21,22]. Related information between features for SED and DOAE may be helpful for the SELD task, which needs to predict the class and direction of a specific sound event simultaneously. The related information can be enhanced by the DCMA using two different embedded features for the queries and keys/values in transformer decoder layers. Therefore, we expect our DCMA-based model structure to efficiently learn the association between the two embedded features.

In addition, the decoder consists of one common block based on the DCMA despite the need for multiple predictions with the track-wise output format. Therefore, an efficient SELD model can be constructed even for the task with polyphonic sound events. We refer to our model as DCMA-SELD.

Unlike speech, in which a feature extraction method suitable for recognition has been established, various feature extraction methods are evaluated for sound signal analysis, and a suitable combination for SELD is found among them. In addition, we try and evaluate data augmentation techniques of the mixup simulating polyphonic events and channel rotation for spatial augmentation to improve the generalization performance of our model.

The core of the model architecture has been evaluated in DCASE2021 Task3 [4], and this paper further develops the model by the methods described above with a detailed description and demonstrates the appropriateness and effectiveness of the model through extensive experiments.

The rest of the paper is organized as follows. In Section 2, we describe our approach including detailed explanations for DCMA-SELD, input features, and data augmentation techniques. Section 3 presents the experimental setup, results, and discussions. Finally, we conclude the paper with some remarks in Section 4.

2. Proposed Approach

This section describes the architecture of our model DCMA-SELD, input features, and data augmentation techniques in detail. The DCMA-SELD is split into two streams: the SED and DOAE streams. The log-mel spectrogram, chromagram, and additional spectral information extracted from four channel signals in the FOA format are used for both streams. Besides, IVs are additionally used for DOAE. Two types of data augmentation techniques have been tried to increase the generalization performance of the model [2,15]. They are the mixup that sums up weighted audio clips to simulate polyphonic events [23] and the channel rotation by exchanging audio channels [24]. These data augmentation techniques are applied to the whole training dataset to train our model.

2.1. Input Features

2.1.1. Log-Mel Spectrogram

A spectrogram is a representation of a frequency-specific value of a signal that changes with time. For analysis in the frequency domain, the time-domain signal is divided into sections by a certain number of data samples corresponding to the frame size. The short-time Fourier transform (STFT) is performed for the data in each section. In general, an appropriate segmentation length is 20–35 ms for the audio analysis. We set the frame and hop sizes for the STFT to 1024 and 600 samples, respectively.

Humans are more sensitive to changes in a low-frequency band than in a high-frequency band. Therefore, it is effective to extract and use the feature vector by focusing on the low-frequency band rather than the entire frequency band. The mel-filter bank consists of filters with narrower bandwidths as the frequency decreases to analyze the change in a relatively low-frequency range in detail. A mel-spectrogram is obtained by analyzing data by a mel-filter bank. In addition, humans notice the intensity difference more sensitively at a lower power level than at a higher power level. To magnify the intensity difference as the power level is lower, a log-mel spectrogram is obtained by taking the logarithmic value on a mel-spectrogram. We set the number of filters to 256 for the mel-spectrogram.

2.1.2. Additional Spectral Information

Various spectral features can be extracted through additional calculations on a spectrogram. In this paper, spectral centroid [25] (1st order), spectral bandwidth [25] (1st order), spectral contrast [26] (7th order), spectral flatness [27] (1st order), and spectral rolloff [28] (1st order) are concatenated to provide 11-dimensional spectral information at a frame. The spectral centroid is a frequency where the center of mass of the spectrum at a frame is located. This has a strong correlation with the brightness of sound [29]. The spectral bandwidth indicates the dispersion of the spectrum that is computed from the spectral centroid. The spectral contrast is the average of the difference between peak and valley energies in each sub-band after dividing the spectrum into many sub-bands [26]. A high-contrast value means a clear narrow-band signal whereas a wide-band noise signal provides a small contrast value. We use six sub-bands in the experiments. The spectral flatness is a measure that quantifies the degree to which the spectrum is similar to white noise [27]. The spectral rolloff provides a frequency at which the spectral energy contained at and below that frequency is a certain percentage of the total energy at a frame.

2.1.3. Chromagram

The chroma vector represents the amount of energy of an audio segment in 12 pitch classes of {C, C#, D, D#, E, F, F#, G, G#, A, A#, B} represented by Western music notation, irrespective of the octave the segment is in [30]. By shifting the time window across the audio signal, a sequence of chroma vectors referred to as a chromagram is obtained to show how the pitch changes over time. In addition, a log chromagram is obtained by taking the logarithmic value on a chromagram. The local chroma features are still too sensitive, particularly when looking at variations in the articulation and local tempo deviations. To obtain robust features, chroma energy normalized statistics (CENS) features represent a kind of weighted statistics of the energy distribution over a window of 41 consecutive chroma vectors [31].

Since the chroma feature is particularly related to the harmonic structure and melody, it is useful for analyzing music. Since target events such as piano, ringing phone, and alarm have musical elements, it is expected to help distinguish them from non-musical events such as human sounds, footsteps, and door knocks.

2.1.4. Intensity Vector

Since the spectral features described above are used for SED, additional features are required for DOAE. Acoustic IVs extracted from the FOA format are used for DOAE because they convey the information about the acoustic energy direction of the sound wave [11]. In the FOA format, omnidirectional and three-directional components are indicated in the order of the first to fourth channels, respectively. An acoustic IV consists of the real parts of the values computed by multiplying the conjugate value of the first channel signal and the value of the remaining channel signals in the STFT domain expressed as [12]

$$\begin{array}{c}\hfill \mathbf{I}(f,t)=\Re \left\{{\mathbf{W}}^{*}(f,t)·\left[\begin{array}{c}\mathbf{X}(f,t)\\ \mathbf{Y}(f,t)\\ \mathbf{Z}(f,t)\end{array}\right]\right\},\end{array}$$

(5)

where $\mathbf{W}(f,t)$, $\mathbf{X}(f,t)$, $\mathbf{Y}(f,t)$, and $\mathbf{Z}(f,t)$ are spectral values of the first to fourth channel signals at frequency bin f and frame t in the STFT domain, respectively. $\Re \{·\}$ and * indicate the real part and the conjugate of a complex-valued matrix, respectively. The resulting three-dimensional vector is normalized by

$$\begin{array}{c}\hfill \overline{\mathbf{I}}(f,t)=\frac{\mathbf{I}(f,t)}{{\parallel \mathbf{I}(f,t)\parallel }_2},\end{array}$$

(6)

where ${\parallel \parallel }_2$ represents the l2-norm.

Since the generalized cross-correlation with phase transform (GCC-PHAT) extracted from the microphone array is the most basic feature in the time difference of arrival (TDOA) problem, many participants used it a lot in previous challenges. However, this feature is noisy when multiple sound sources overlap or a moving source occurs. Since the TNSSE 2021 dataset contains the events moving at an angular speed of 40 degrees per second in addition to overlapped sound sources, DOAE may not be reliable using the GCC-PHAT. To effectively solve SELD in the multi-source scenario with interference, only the IV without the GCC-PHAT is used for DOAE in this paper.

2.2. Data Augmentation Techniques

2.2.1. Mixup

Since there are polyphonic events in the task, it is useful to make situations with overlapped events. A mixed datum is created by a weighted sum of two randomly selected data. In the original mixup used for image classification [23], the weight value multiplied by the data is also applied to the labels as in the formula below:

$$\begin{array}{ccc}\hfill {\mathbf{x}}^{\prime }& =& \lambda {\mathbf{x}}_i+(1-\lambda ){\mathbf{x}}_j,\hfill \end{array}$$

(7)

$$\begin{array}{ccc}\hfill y^{\prime }& =& \lambda y_i+(1-\lambda )y_j,\hfill \end{array}$$

(8)

where ${\mathbf{x}}_i$ and $y_i$ indicate the i-th data and label, respectively. The value of the weight $\lambda$ is randomly selected from a uniform distribution between 0.5 and 0.8 and applied for the mixup.

In the SED task, all the mixed events should be detected even though any one of those events is mixed with small power. Therefore, in this paper, $\lambda$ is applied only to the data, and the corresponding labels are simply added.

2.2.2. Augmentation by Rotation

Because data captured from a microphone array are usually limited, data augmentation for multi-channel data is very important for deep learning. The spatial augmentation method in [32] is a very effective augmentation method that can be used in the FOA format to increase the numbers of DOA labels without losing the physical relationships between steering vectors and observations. Since the spatial responses of the four FOA channels $\{\mathbf{W},\mathbf{X},\mathbf{Y},\mathbf{Z}\}$ to a source incident from a direction are given by Equations (1)–(4), data for a new direction can be reproduced by the reflection on a specific plane or the rotation of the angle by $\pm {90}^{°}$ or ${180}^{°}$. Whereas azimuths span the whole range of $\varphi \in [-{180}^{°},{180}^{°})$, elevations span a partial range of $\theta \in [{-45}^{°},{45}^{°}]$. In order not to go out of the label range, only the reflection is applied for the elevation while the reflection and rotations of $\pm {90}^{°}$ can be applied for the azimuth. Then, a total of 16 combinations of rotations are applicable for augmentation, as summarized in

Table 1. Sixteen cases of spatial augmentation. The channel $\mathbf{W}$ carrying omnidirectional information does not change during the augmentation.

Case	$\mathit{\theta }$	$-\mathit{\theta }$
$\mathbf{\varphi }$−${\mathbf{90}}^{°}$	${\mathbf{X}}^{{}^{\prime }}\leftarrow \mathbf{Y}$, ${\mathbf{Y}}^{{}^{\prime }}\leftarrow -\mathbf{X}$, ${\mathbf{Z}}^{{}^{\prime }}\leftarrow \mathbf{Z}$	${\mathbf{X}}^{{}^{\prime }}\leftarrow ,\mathbf{Y}$${\mathbf{Y}}^{{}^{\prime }}\leftarrow -\mathbf{X}$, ${\mathbf{Z}}^{{}^{\prime }}\leftarrow -\mathbf{Z}$
$\mathbf{\varphi }$	${\mathbf{X}}^{{}^{\prime }}\leftarrow \mathbf{X}$, ${\mathbf{Y}}^{{}^{\prime }}\leftarrow \mathbf{Y}$, ${\mathbf{Z}}^{{}^{\prime }}\leftarrow \mathbf{Z}$	${\mathbf{X}}^{{}^{\prime }}\leftarrow \mathbf{X}$, ${\mathbf{Y}}^{{}^{\prime }}\leftarrow \mathbf{Y}$, ${\mathbf{Z}}^{{}^{\prime }}\leftarrow -\mathbf{Z}$
$\mathbf{\varphi }$ + ${\mathbf{90}}^{°}$	${\mathbf{X}}^{{}^{\prime }}\leftarrow -\mathbf{Y}$, ${\mathbf{Y}}^{{}^{\prime }}\leftarrow \mathbf{X}$, ${\mathbf{Z}}^{{}^{\prime }}\leftarrow \mathbf{Z}$	${\mathbf{X}}^{{}^{\prime }}\leftarrow -\mathbf{Y}$, ${\mathbf{Y}}^{{}^{\prime }}\leftarrow \mathbf{X}$, ${\mathbf{Z}}^{{}^{\prime }}\leftarrow -\mathbf{Z}$
$\mathbf{\varphi }$ + ${\mathbf{180}}^{°}$	${\mathbf{X}}^{{}^{\prime }}\leftarrow -\mathbf{X}$, ${\mathbf{Y}}^{{}^{\prime }}\leftarrow -\mathbf{Y}$, ${\mathbf{Z}}^{{}^{\prime }}\leftarrow \mathbf{Z}$	${\mathbf{X}}^{{}^{\prime }}\leftarrow -\mathbf{X}$, ${\mathbf{Y}}^{{}^{\prime }}\leftarrow -\mathbf{Y}$, ${\mathbf{Z}}^{{}^{\prime }}\leftarrow -\mathbf{Z}$
$-\mathbf{\varphi }$−${\mathbf{90}}^{°}$	${\mathbf{X}}^{{}^{\prime }}\leftarrow \mathbf{Y}$, ${\mathbf{Y}}^{{}^{\prime }}\leftarrow -\mathbf{X}$, ${\mathbf{Z}}^{{}^{\prime }}\leftarrow \mathbf{Z}$	${\mathbf{X}}^{{}^{\prime }}\leftarrow -\mathbf{Y}$, ${\mathbf{Y}}^{{}^{\prime }}\leftarrow -\mathbf{X}$, ${\mathbf{Z}}^{{}^{\prime }}\leftarrow -\mathbf{Z}$
$-\mathbf{\varphi }$	${\mathbf{X}}^{{}^{\prime }}\leftarrow \mathbf{X}$, ${\mathbf{Y}}^{{}^{\prime }}\leftarrow -\mathbf{Y}$, ${\mathbf{Z}}^{{}^{\prime }}\leftarrow \mathbf{Z}$	${\mathbf{X}}^{{}^{\prime }}\leftarrow \mathbf{X}$, ${\mathbf{Y}}^{{}^{\prime }}\leftarrow -\mathbf{Y}$, ${\mathbf{Z}}^{{}^{\prime }}\leftarrow -\mathbf{Z}$
$-\mathbf{\varphi }$ + ${\mathbf{90}}^{°}$	${\mathbf{X}}^{{}^{\prime }}\leftarrow \mathbf{Y}$, ${\mathbf{Y}}^{{}^{\prime }}\leftarrow \mathbf{X}$, ${\mathbf{Z}}^{{}^{\prime }}\leftarrow \mathbf{Z}$	${\mathbf{X}}^{{}^{\prime }}\leftarrow \mathbf{Y}$, ${\mathbf{Y}}^{{}^{\prime }}\leftarrow \mathbf{X}$, ${\mathbf{Z}}^{{}^{\prime }}\leftarrow -\mathbf{Z}$
$-\mathbf{\varphi }$ + ${\mathbf{180}}^{°}$	${\mathbf{X}}^{{}^{\prime }}\leftarrow -\mathbf{X}$, ${\mathbf{Y}}^{{}^{\prime }}\leftarrow \mathbf{Y}$, ${\mathbf{Z}}^{{}^{\prime }}\leftarrow \mathbf{Z}$	${\mathbf{X}}^{{}^{\prime }}\leftarrow -\mathbf{X}$, ${\mathbf{Y}}^{{}^{\prime }}\leftarrow \mathbf{Y}$, ${\mathbf{Z}}^{{}^{\prime }}\leftarrow -\mathbf{Z}$

.

2.3. Model Architecture

2.3.1. Encoder

Input features are effectively embedded by applying convolutional blocks and parameter-sharing techniques in the encoder layers. In the last decades, deep learning with convolutional neural networks (CNNs) at the forefront has dramatically upgraded the performance in several research fields (image classification, object detection, optical character recognition (OCR), etc.). CNN-based models such as AlexNet, GoogleNet, VGGNet, Xception, ResNet, ResNeXt, and DenseNet have been released [33]. Especially, its efficient representation directly extracted from raw input data based on these verified models leads to impressive performance. In this study, to extract rich internal features, local correlation information is obtained from acoustic features through convolutional layers.

Multi-task learning techniques have been successfully applied to various tasks such as natural language processing (NLP), automatic speech recognition (ASR), and computer vision [34]. Predicting the class and direction of a sound event are different tasks. However, they are closely related to each other, since they originate from the same sound. No matter how noisy the environment is, if someone focuses on a specific target sound, he can catch the direction of the sound source. Likewise, if he concentrates on a particular direction, it becomes easier to tell what sound is coming from that direction. Therefore, SELD is an appropriate task for multi-task learning. In the SELD, a no-parameter sharing scheme where features for SED and DOAE are not shared in any embedding stage cannot exploit the association between the SED and DOAE features. All of the parameters are shared between two subtasks in a hard parameter-sharing scheme. It can reduce the overfitting phenomenon to a specific subtask, but it can cause performance degradation because it cannot discard the feature that is only necessary for just one subtask [35]. Soft parameter sharing has a separate set of parameters per subtask and only shares partial instances, which are a crucial part of the learning. It is expected that only the information useful for each subtask will be complemented [13].

Although we use additional spectral information and chromagram concatenated with the log-mel spectrogram as the spectral feature, the embedding scheme of the EINV2 corresponding to four convolutional blocks with batch normalization (BN), ReLU activation, average pooling, and soft parameter sharing [13] is adopted for the encoder part since it is an encoding scheme that sufficiently reflects the characteristics of SELD discussed above.

2.3.2. Decoder

The decoder part of the transformer is modified to efficiently decode the information in the embedded features for the prediction of sound events or source directions. In particular, the prediction can be improved by exploiting the related information from the other embedded features using the DCMA in the decoder. The overall schematic diagram of the decoder is represented in Figure 1.

Similar to the encoder, the decoder is also split into SED and DOAE streams, each of which is based on the decoder part of the transformer, which was first proposed by Vaswani et al. [20]. Although the transformer composed of attention mechanisms without convolution or recurrent layers has used the previous decoder output as the current decoder input to analyze sequence data [20], our decoder uses the embedded features for either SED or DOAE as the decoder input to analyze the current sound directly.

As shown in Figure 1, the decoder in each stream is composed of a stack of identical layers. Each layer has two multi-head attention blocks followed by an FC network. The first attention block is the masked multi-head self-attention block using the common input embedding as the query, key, and value, while the second attention block is the multi-head attention based on the DCMA with embedded features for SED and DOAE (corresponding to the outputs of the first attention block) used as the queries and keys/values. Residual connections are employed around each of the blocks and the FC network, followed by layer normalization.

Multi-head attention is a module for attention mechanisms which runs through an attention mechanism several times in parallel [20]. Then, independent attention outputs are concatenated and linearly transformed into the original dimension. Therefore, multiple attention heads focus on different aspects of input features during the attention process, and the outputs are concatenated to enable effective attention. In addition, since each attention mechanism is performed in a small dimension, it is possible to benefit from the amount of computation. After passing through the multi-head self-attention that is also employed in the EINV2 [13], we exchange the related information from the other features embedded through the multi-head self-attention in the two streams by applying the DCMA [21] instead of the encoder–decoder multi-head attention in the transformer. Although the DCMA was introduced to utilize both an audio context vector using video query and a video context vector using audio query in audio-visual speech recognition [21], the related information between the embedded features for SED or DOAE can be enhanced in learning the association between the two features by employing the DCMA architecture with the two different features used as the queries and keys/values. Using the scaled dot-product attention, attention outputs in the i-th head, ${\mathbf{A}}_{\mathrm{SD}}$ and ${\mathbf{A}}_{\mathrm{DS}}$, can be expressed as

$$\begin{array}{ll}{\mathbf{A}}_{\mathrm{SD}}& =\mathrm{Attention}({\mathbf{E}}_{\mathrm{S}},{\mathbf{E}}_{\mathrm{D}},{\mathbf{E}}_{\mathrm{D}})\\ & =\mathrm{softmax}\left(\frac{{\mathbf{E}}_{\mathrm{S}}{\mathbf{W}}_{\mathrm{S}}^{{\mathrm{Q}}_i}{\left({\mathbf{W}}_{\mathrm{S}}^{{\mathrm{K}\mathrm{V}}_i}\right)}^{\mathrm{T}}{\mathbf{E}}_{\mathrm{D}}^{\mathrm{T}}}{\sqrt{1/d_k}}\right){\mathbf{E}}_{\mathrm{D}}{\mathbf{W}}_{\mathrm{S}}^{{\mathrm{K}\mathrm{V}}_i},\end{array}$$

(9)

$$\begin{array}{ll}{\mathbf{A}}_{\mathrm{DS}}& =\mathrm{Attention}({\mathbf{E}}_{\mathrm{D}},{\mathbf{E}}_{\mathrm{S}},{\mathbf{E}}_{\mathrm{S}})\\ & =\mathrm{softmax}\left(\frac{{\mathbf{E}}_{\mathrm{D}}{\mathbf{W}}_{\mathrm{D}}^{{\mathrm{Q}}_i}{\left({\mathbf{W}}_{\mathrm{D}}^{{\mathrm{K}\mathrm{V}}_i}\right)}^{\mathrm{T}}{\mathbf{E}}_{\mathrm{S}}^{\mathrm{T}}}{\sqrt{1/d_k}}\right){\mathbf{E}}_{\mathrm{S}}{\mathbf{W}}_{\mathrm{D}}^{{\mathrm{K}\mathrm{V}}_i},\end{array}$$

(10)

where ${\mathbf{E}}_{\mathrm{S}}\in {\mathbb{R}}^{N_T\times D_F}$ and ${\mathbf{E}}_{\mathrm{D}}\in {\mathbb{R}}^{N_T\times D_F}$ are SED and DOAE features used as inputs with $N_T$ and $D_F$ denoting the number of time steps and the feature dimension, respectively. ${\mathbf{W}}_{\mathrm{S}}^{{\mathrm{Q}}_i}$ and ${\mathbf{W}}_{\mathrm{S}}^{{\mathrm{K}\mathrm{V}}_i}$ represent learnable query and key/value matrices for the i-th head in the SED stream, respectively. In the same way, ${\mathbf{W}}_{\mathrm{D}}^{{\mathrm{Q}}_i}$ and ${\mathbf{W}}_{\mathrm{D}}^{{\mathrm{K}\mathrm{V}}_i}$ denote the corresponding matrices in the DOAE stream. $d_k$ is the common number of columns of these learnable matrices.

Experiments were conducted by changing the number of decoder layers from one to six, and the best results were obtained when set to two. In addition to the soft parameter sharing in the encoder part, the DCMA process will play a major role in obtaining the association between SED and DOAE feature information prior to the prediction.

2.3.3. Overall Procedure

The overall DCMA-SELD architecture is illustrated in Figure 2, and the feature dimensions and hyper-parameters are summarized in

Table 2. Feature dimensions and hyper-parameters of DCMA-SELD. ${\alpha }^i$, ${\beta }^i$, ${\gamma }^i$, and ${\delta }^i$ denote the learnable weights for the soft parameter sharing on the input embeddings for the i-th convolutional block.

		DOAE Stream	SED Stream
Input	Feature Extraction DOAE: B × 7 × 160 × 279 SED: B × 4 × 160 × 279	(4 × 279) - Log-mel (256-dim) - Add. Spectral info.(11-dim) - Chroma (12-dim) + (3 × 279) - IV (256-dim) - Zero vector (23-dim)	(4 × 279) - Log-mel (256-dim) - Add. Spectral info.(11-dim) - Chroma (12-dim)
Encoder	Conv. Block 1 DOAE: B × 64 × 80 × 139 SED: B × 64 × 80 × 139	Conv2d (3 × 3, 4 → 64) BN (64) ReLU Conv2d (3 × 3, 64 → 64) BN (64) ReLU	Conv2d (3 × 3, 4 → 64) BN (64) ReLU Conv2d (3 × 3, 64 → 64) BN (64) ReLU
		Average Pooling (2 × 2)	Average Pooling (2 × 2)
	Soft parameter sharing 1	Learnable weights ${\mathsf{\alpha }}^1$(64) Learnable weights ${\mathsf{\beta }}_{}^1$ (64)	Learnable weights ${\mathsf{\gamma }}_{}^1$ (64) Learnable weights ${\mathsf{\delta }}_{}^1$(64)
	Conv. Block 2 DOAE: B × 128 × 40 × 69 SED: B × 128 × 40 × 69	Conv2d (3 × 3, 64 → 128) BN (128) ReLU Conv2d (3 × 3, 128 → 128) BN (128) ReLU	Conv2d (3 × 3, 64 → 128) BN (128) ReLU Conv2d (3 × 3, 128 → 128) BN (128) ReLU
		Average Pooling (2 × 2)	Average Pooling (2 × 2)
	Soft parameter sharing 2	Learnable weights ${\mathsf{\alpha }}^2$(128) Learnable weights ${\mathsf{\beta }}_{}^2$(128)	Learnable weights ${\mathsf{\gamma }}_{}^2$(128) Learnable weights ${\mathsf{\delta }}_{}^2$(128)
	Conv. Block 3 DOAE: B × 256 × 40 × 34 SED: B × 256 × 40 × 34	Conv2d (3 × 3, 128 → 256) BN (256) ReLU Conv2d (3 × 3, 256 → 256) BN (256) ReLU	Conv2d (3 × 3, 128 → 256) BN (256) ReLU Conv2d (3 × 3, 256 → 256) BN (256) ReLU
		Average Pooling (1 × 2)	Average Pooling (1 × 2)
	Soft parameter sharing 3	Learnable weights ${\mathsf{\alpha }}^3$(256) Learnable weights ${\mathsf{\beta }}_{}^3$(256)	Learnable weights ${\mathsf{\gamma }}_{}^3$(256) Learnable weights ${\mathsf{\delta }}_{}^3$(256)
	Conv. Block 4 DOAE: B × 512 × 40 × 17 SED: B × 512 × 40 × 17	Conv2d (3 × 3, 256 → 512) BN (512) ReLU Conv2d (3 × 3, 512 → 512) BN (512) ReLU	Conv2d (3 × 3, 256 → 512) BN (512) ReLU Conv2d (3 × 3, 512 → 512) BN (512) ReLU
		Average Pooling (1 × 2)	Average Pooling (1 × 2)
Average Pooling	Average Pooling DOAE: B × 512 × 40 SED: B × 512 × 40	Average Pooling ( @F )	Average Pooling ( @F )
Decoder	Transformer decoder ( × 2layers) DOAE: B × 512 × 40 SED: B × 512 × 40	Masked MHSA (512, heads=8) LayerNorm (512) DCMA (512 → 512, heads=8) LayerNorm (512) FC(512 → 2048) FC(2048 → 512) LayerNorm (512)	Masked MHSA (512, heads=8) LayerNorm (512) DCMA (512 → 512, heads=8) LayerNorm (512) FC(512 → 2048) FC(2048 → 512) LayerNorm (512)
Prediction Layer	DOAE: B × 3 × 40 SED: B × 12 × 40	FC1 (512 → 3 ), FC2 (512 → 3 ), FC3 (512 → 3 )	FC1 (512 → 12 ), FC2 (512 → 12 ), FC3 (512 → 12 )

.

For the SED stream, the input feature tensor has four channels. Each of the four channels has spectral features obtained from each of the four signals formatted in the FOA format, which consists of a 256-dimensional log-mel spectrum, a 12-dimensional chroma vector, and an 11-dimensional spectral feature per frame. For the DOAE stream, the input feature tensor has three more channels in addition to the four channels for the SED stream. Each of the three channels has a 256-dimensional IV concatenated with a 23-dimensional zero vector to match the dimension by zero padding per frame.

The SED and DOAE streams have the same encoder architecture, which consists of four convolutional blocks, and a layer of CNN, BN, and ReLU activation is repeated twice in each block. After an embedding passes a block followed by average pooling in each stream, the resulting embeddings in both the streams are combined by soft parameter sharing to provide embeddings input to the next block, which are expressed as

$$\begin{array}{ccc}\hfill {\overline{\mathbf{D}}}_c^{(i+1)}& =& {\alpha }_c^{\left(i\right)}{\mathbf{D}}_c^{\left(i\right)}+{\beta }_c^{\left(i\right)}{\mathbf{S}}_c^{\left(i\right)},\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hfill {\overline{\mathbf{S}}}_c^{(i+1)}& =& {\gamma }_c^{\left(i\right)}{\mathbf{D}}_c^{\left(i\right)}+{\delta }_c^{\left(i\right)}{\mathbf{S}}_c^{\left(i\right)},\hfill \end{array}$$

(12)

where ${\overline{\mathbf{D}}}_c^{\left(i\right)}$ and ${\overline{\mathbf{S}}}_c^{\left(i\right)}$ denote the input embeddings for the i-th convolutional block at the c-th channel in the DOAE and SED streams, respectively, while ${\mathbf{D}}_c^{\left(i\right)}$ and ${\mathbf{S}}_c^{\left(i\right)}$ denote the corresponding output embeddings from a block followed by average pooling, respectively. ${\mathit{\alpha }}^{\left(i\right)}=[{\alpha }_1^{\left(i\right)},{\alpha }_2^{\left(i\right)},...,{\alpha }_C^{\left(i\right)}]$, ${\mathit{\beta }}^{\left(i\right)}=[{\beta }_1^{\left(i\right)},{\beta }_2^{\left(i\right)},...,{\beta }_C^{\left(i\right)}]$, ${\mathit{\gamma }}^{\left(i\right)}=[{\gamma }_1^{\left(i\right)},{\gamma }_2^{\left(i\right)},...,{\gamma }_C^{\left(i\right)}]$, and ${\mathit{\delta }}^{\left(i\right)}=[{\delta }_1^{\left(i\right)},{\delta }_2^{\left(i\right)},...,{\delta }_C^{\left(i\right)}]$ are learnable weights applied to the output embeddings for the i-th convolutional block in the soft parameter sharing, where C is the number of channels.

In each stream, the encoder output is input to the decoder based on the decoder part of the transformer, as mentioned above. The decoder in each stream is composed of a stack of two identical layers, and each layer has a masked multi-head self attention block and a multi-head attention based on the DCMA followed by an FC network. In particular, the second attention block utilizes embedded features for SED and DOAE used as the queries and keys/values to enhance the related information between the features.

The decoder is followed by an FC layer to predict SELD outputs. Different from the class-wise output format, the track-wise output format can detect sound events of the identical class from different directions, which is called the case of homogeneous overlap [13]. Since the DCASE2021 Task3 may have up to three overlapping events, we adopt the track-wise output format [13] where three SED outputs and three DOAE outputs should be given by the FC layer. In either the SED or DOAE stream, three FC blocks in parallel simply pass a decoder output to predict three SED or DOAE outputs. Our system adopting the track-wise output format is trained by the permutation-invariant training for each frame [13].

3. Experiments and Results

3.1. Experimental Setup

3.1.1. Parameter Configuration

The sampling rate of the audio signals is 32 kHz. The frame and hop sizes for the STFT were set to 1024 and 600 samples, respectively. Spectral features composed of log-mel spectrogram, chromagram, and additional spectral information were obtained by librosa [28,36]. An Adam optimizer [37] was used for the training. The StepLR scheduler [38] that decays the learning rate by a factor every configured epoch was applied to control the learning rate during the training. For our experiments, we decayed the learning rate by a factor of 0.1 every 80 epochs. The initial learning rate was set to 0.0005.

3.1.2. Evaluation Metrics

The performance of the model was assessed by the evaluation metrics used in the DCASE2021 Challenge, which consider joint dependencies between the performance of the SED and DOAE. In the SELD, the F1-score $F_{\le }{}_T$ and error rate $E{R}_{\le }{}_T$ for SED are location-dependent, considering true positives for the predictions only when the difference between the DOAE and the reference is under a threshold T. Here, T was set to ${20}^{°}$. For the metrics of DOAE, localization error $L{E}_{\mathrm{CD}}$ and localization recall $L{R}_{\mathrm{CD}}$ are designed class-dependent, meaning that they are computed only across each class and then averaged. $L{E}_{\mathrm{CD}}$ shows the average angular distance between predictions and references of the same class, whereas $L{R}_{\mathrm{CD}}$ calculates the true positive rate of the localization predictions in a class out of all class instances [8].

These metrics can be integrated into a joint measurement $D_{\mathrm{SELD}}$ as given in [17]:

$$D_{\mathrm{SELD}}=\frac{1}{4}[E{R}_{{\le }_T}+(1-F_{{\le }_T})+\frac{L{E}_{\mathrm{CD}}}{{180}^{°}}+(1-L{R}_{\mathrm{CD}})].$$

(13)

The higher $F_{{\le }_T}$ and $L{R}_{\mathrm{CD}}$ and the lower $E{R}_{{\le }_T}$, $L{E}_{\mathrm{CD}}$, and $D_{\mathrm{SELD}}$ means better performance.

3.2. Experimental Results and Discussions

Since the metadata information of the evaluation dataset is not provided even after the challenge, we cannot check the performance on the evaluation dataset. Instead, we compared the performance of the models on the test set in the development dataset. For all our results in the following tables, we included the 95% confidence intervals for each metric, which were estimated using the jackknife procedure, with the partial estimates calculated in a leave-one-out manner, excluding, in turns, one audio file from the test set, as presented in [8,39].

Table 3. Ablation study on input features for our DCMA-SELD on the TNSSE 2021 development dataset. All data represent mean values and associated 95% confidence intervals. The highest performance in each metric is indicated in bold.

Feature	${\mathit{ER}}_{{\le }_{{20}^{°}}}$	${\mathit{F}}_{{\le }_{{20}^{°}}}$ (%)	${\mathit{LE}}_{\mathbf{CD}}$ (${}^{°}$)	${\mathit{LR}}_{\mathbf{CD}}$ (%)	${\mathit{D}}_{\mathbf{SELD}}$
mel sp.+IV	0.44 ± 0.02	64.8 ± 1.9	14.4 ± 0.8	79.1 ± 1.9	0.269 ± 0.014
mel sp.+add. sp. info.+IV	0.44 ± 0.02	64.2 ± 2.1	14.8 ± 0.7	78.8 ± 2.1	0.273 ± 0.016
mel sp.+chroma+IV	0.42 ± 0.02	65.2 ± 2.1	14.0 ± 0.7	78.5 ± 1.9	0.267 ± 0.015
mel sp.+add. sp. info.+chroma+IV	0.41 ± 0.02	67.1 ± 2.1	13.6 ± 0.7	79.0 ± 2.0	0.256 ± 0.016

summarizes an ablation study on input features for our DCMA-SELD. Retaining the log-mel spectrogram and IVs as basic features for SED and DOAE, performance was evaluated when some of the four types of input features were dropped. When all four kinds of features were applied, the best results were obtained in all the metrics except for $L{R}_{\mathrm{CD}}$, which was 0.1% lower than the best score. The performance was degraded without some of them, which demonstrates that all types of features contribute to performance improvement. This is because the log-mel spectrogram, additional spectral information, and chroma features are obtained from spectra with different frequency resolutions and thus contain information on different aspects of the input audio signal. In the following experiments, all four types of features were used.

Table 4. Performance for different parameter-sharing methods in the encoder of our DCMA-SELD on the TNSSE 2021 development dataset. All data represent mean values and associated 95% confidence intervals. The highest performance in each metric is indicated in bold.

Parameter Sharing	${\mathit{ER}}_{{\le }_{{20}^{°}}}$	${\mathit{F}}_{{\le }_{{20}^{°}}}$ (%)	${\mathit{LE}}_{\mathbf{CD}}$ (${}^{°}$)	${\mathit{LR}}_{\mathbf{CD}}$ (%)	${\mathit{D}}_{\mathbf{SELD}}$
No parameter sharing	0.46 ± 0.02	63.2 ± 1.9	14.3 ± 0.8	75.4 ± 2.1	0.287 ± 0.014
Hard parameter sharing	0.44 ± 0.02	65.6 ± 2.0	13.6 ± 0.7	75.9 ± 2.0	0.278 ± 0.015
Soft parameter sharing	0.41 ± 0.02	67.1 ± 2.1	13.6 ± 0.7	79.0 ± 2.0	0.256 ± 0.016

compares the performance when using different parameter-sharing methods in the encoder of our DCMA-SELD model. As in [13], our model also showed the best performance for all given official metrics with soft parameter sharing. On the other hand, the worst performance was recorded for all the metrics when the SED and DOAE embeddings were performed independently with no parameter sharing. Through this experiment, it can be confirmed that the two embedded features need not be completely identical but are related and complementary.

Table 5. Performance for different decoder architectures in our DCMA-SELD on the TNSSE 2021 development dataset. The numbers in the parentheses indicate the numbers of decoder blocks used in the models to perform three predictions. All data represent mean values and associated 95% confidence intervals. The highest performance in each metric is indicated in bold.

Decoder Architecture	Number of Parameters	${\mathit{ER}}_{{\le }_{{20}^{°}}}$	${\mathit{F}}_{{\le }_{{20}^{°}}}$ (%)	${\mathit{LE}}_{\mathbf{CD}}$ (${}^{°}$)	${\mathit{LR}}_{\mathbf{CD}}$ (%)	${\mathit{D}}_{\mathbf{SELD}}$
LSTM (3)	34,617,325	0.49 ± 0.02	59.2 ± 2.1	17.5 ± 0.8	76.2 ± 2.2	0.309 ± 0.016
Multi-head self-attention (3)	34,635,757	0.42 ± 0.02	66.7 ± 2.1	13.1± 0.7	78.0 ± 2.1	0.260 ± 0.016
Multi-head self-attention (1)	17,813,485	0.45 ± 0.02	63.0 ± 2.2	15.8 ± 0.9	77.5 ± 2.2	0.283 ± 0.017
DCMA-SELD (1)	26,218,477	0.41 ± 0.02	67.1 ± 2.1	13.6 ± 0.7	79.0 ± 2.0	0.256 ± 0.016

shows the performance when the decoder of our model was replaced with different architectures. As a basic decoder to deal with audio data, two-layer LSTM networks were evaluated. In each stream, three LSTM blocks in parallel were used to predict SED or DOAE outputs in this task where up to three overlapping events may occur. In addition, the decoder was also replaced with the multi-head self-attention, which corresponded to the model in [13]. Since the model in [13] used one multi-head self-attention block for each track, three blocks were used in the task. On the other hand, even with three tracks, our model obtained three parallel prediction results from an FC layer after passing through one common decoder. Therefore, instead of using one multi-head self-attention block per track, we also evaluated the model performing three predictions after one common multi-head self-attention block.

The LSTM-based decoder showed the lowest performance in all metrics. When the multi-head self-attention blocks were used as the decoder, the case of using three blocks achieved higher performance in all metrics than the case of using one common block. However, except for $L{E}_{\mathrm{CD}}$, which provided a greater error by 0.5°, the decoder used in our DCMA-SELD outperformed the other decoder architectures despite using one common decoder block. This demonstrates that building an efficient model through one common decoder block which can produce sufficiently reliable outputs is more effective in improving generalization performance than using independent decoder blocks with too many weights in this task that has to predict three tracks.

Table 6. Performance of our DCMA-SELD models trained with and without data augmentation for model training on the TNSSE 2021 development dataset. All data represent mean values and associated 95% confidence intervals. The highest performance in each metric is indicated in bold.

Augmentation	${\mathit{ER}}_{{\le }_{{20}^{°}}}$	${\mathit{F}}_{{\le }_{{20}^{°}}}$ (%)	${\mathit{LE}}_{\mathbf{CD}}$ (${}^{°}$)	${\mathit{LR}}_{\mathbf{CD}}$ (%)	${\mathit{D}}_{\mathbf{SELD}}$
Without augmentation	0.50 ± 0.02	58.6 ± 1.9	17.9 ± 0.9	76.5 ± 1.8	0.312 ± 0.014
With augmentation	0.41 ± 0.02	67.1 ± 2.1	13.6 ± 0.7	79.0 ± 2.0	0.256 ± 0.016

compares the performance of the models trained with and without data augmentation for model training described in Section 2.2. The model to which the data augmentation was applied improved the performance in all metrics, which showed that the data augmentation was very useful in the task.

Table 7. Performance of our DCMA-SELD and the models submitted to the DCASE2021 challenge on the TNSSE 2021 development dataset. The baseline system in the challenge corresponds to the modified version of the SELDnet in [40]. The numbers after # denote the rank of the model in the challenge. “Rank sum” represents the sum of the rankings of the four metrics used as the challenge ranking evaluation method. The highest performance in each metric is indicated in bold.

System	Feature	Format	${\mathit{ER}}_{{\le }_{{20}^{°}}}$	${\mathit{F}}_{{\le }_{{20}^{°}}}$ (%)	${\mathit{LE}}_{\mathbf{CD}}$ (${}^{°}$)	${\mathit{LR}}_{\mathbf{CD}}$ (%)	${\mathit{D}}_{\mathbf{SELD}}$	Rank Sum
Baseline	mel sp.+IV	FOA	0.73	30.7	24.5	44.8	0.528	28
(#1) Shimada et al. [15]	magnitude sp. + PCEN sp. + IPD	FOA	0.43	69.9	11.1	73.2	0.265	10
(#2) Nguyen et al. [17]	eigenvector-aug. log-sp. + mel sp.	FOA	0.37	73.7	11.2	74.1	0.239	7
(#3) Emmanuel et al. [41]	mel sp.+IV	FOA	0.65	57.1	17.5	62.8	0.387	24
(#5) Park et al. [19]	mel sp.+IV	Both	0.44	69.6	13.7	74.2	0.270	13
(#6) Zhang et al. [42]	mel sp.+IV + GCC-PHAT	Both	0.46	63.2	13.9	62.9	0.319	20
DCMA-SELD	mel sp.+add. sp.+chroma+IV	FOA	0.41 ± 0.02	67.1 ± 2.1	13.6 ± 0.7	79.0 ± 2.0	0.256 ± 0.016	10

compares the performance on the development dataset of our model and highly ranked systems in the challenge (note that the presented model in this paper is an improved version of our previous model that placed fourth in the challenge as mentioned in the introduction). In the experimental results, the baseline system in the challenge corresponding to the modified version of the SELDnet in [40] showed significantly inferior performance in all metrics than the other methods, while the others showed relatively comparable performance. Especially, our model achieved the best performance for the metric $L{R}_{\mathrm{CD}}$ by recording the value of $79.0\%$. Excluding our model, Park et al. [19] showed the highest score of $74.2\%$ in $L{R}_{\mathrm{CD}}$ among all the participants of the challenge, which was about $5\%$ lower than ours. Moreover, both no parameter-sharing and hard parameter-sharing methods based on our model in Table 4 still showed scores higher than $75\%$ in $L{R}_{\mathrm{CD}}$, which was better than the other participants.

It is analyzed that the primary reason why our model can achieve overwhelming results in $L{R}_{\mathrm{CD}}$ compared to the others is that we applied the track-wise output format instead of the class-wise output format adopted in the others (the system in [15] is affected by class-wise predictions because it uses both the output formats together). The class-wise output format can only predict up to one event per class at a time. That is, if two or three events of the same class occur at the same time, one and two sound events, respectively, must be abandoned. However, since the track-wise output format predicts one sound event and direction per track, it is possible to cope even with the case where two or more events belonging to the same class occur at the same time. In the TNSSE 2021 dataset, the case where multiple events belonging to the same class occurs simultaneously reaches $10.45\%$ [17], and the maximum possible score of $L{R}_{\mathrm{CD}}$ with ideal SED condition is $92.5\%$ for the system adopting the class-wise output format [19]. In the DCASE2021 challenge, all the highly ranked systems compared here used the class-wise output format despite the handicap to prevent the possibility of mispredicting non-existent overlapped events that could cause performance degradation in other metrics.

Above all, in spite of the possibility of the misprediction, our model ranked second in terms of both $D_{\mathrm{SELD}}$ and the challenge ranking evaluation method determined by the sum of the rankings of the four metrics while retaining a high $L{R}_{\mathrm{CD}}$ with the track-wise output format that could cope even with overlapped sound events. Therefore, combining our model with another model adopting the class-wise output format that can compensate for the possible misprediction of ours can be a promising ensemble method.

4. Conclusions

In this paper, we presented a model using DCMA and soft parameter sharing to simultaneously detect and localize sound events. The CNN-based encoder with parameter sharing exchanges intermediate features in the CNN layers for the SED and DOAE, and the DCMA-based decoder efficiently outputs three predictions to detect overlapping events in the DCASE2021 Task3. In particular, key and value vectors for either SED or DOAE stream in the DCMA were given from the other stream to efficiently learn the association between SED and DOAE features. Acoustic features, such as additional spectral information and chroma features, that had not been commonly used in the SELD task were additionally adopted to improve the performance. Experimental results demonstrated that our efficient model using one common decoder block based on the DCMA was effective for the SELD task with up to three overlapping events while retaining a high $L{R}_{\mathrm{CD}}$ with the track-wise output format. As a future work, we will study prediction rules to induce non-independent and reliable results between the FC blocks. For further improvement, we plan to develop an ensemble network including the presented model.