Effective Sample Selection and Enhancement of Long Short-Term Dependencies in Signal Detection: HDC-Inception and Hybrid CE Loss

Abstract

Signal detection and classification tasks, especially in the realm of audio, suffer from difficulties in capturing long short-term dependencies and effectively utilizing samples. Firstly, audio signal detection and classification need to classify audio signals and detect their onset and offset times; therefore, obtaining long short-term dependencies is necessary. The methods based on RNNs have high time complexity and dilated convolution-based methods suffer from the “gridding issue” challenge; thus, the HDC-Inception module is proposed to efficiently extract long short-term dependencies. Combining the advantages of the Inception module and a hybrid dilated convolution (HDC) framework, the HDC-Inception module can both alleviate the “gridding issue” and obtain long short-term dependencies. Secondly, datasets have large numbers of silent segments and too many samples for some signal types, which are redundant and less difficult to detect, and, therefore, should not be overly prioritized. Thus, selecting effective samples and guiding the training based on them is of great importance. Inspired by support vector machine (SVM), combining soft margin SVM and cross-entropy loss (CE loss), the soft margin CE loss is proposed. Soft margin CE loss can adaptively select support vectors (effective samples) in datasets and guide training based on the selected samples. To utilize datasets more sufficiently, a hybrid CE loss is proposed. Using the benefits of soft margin CE loss and CE loss, hybrid CE loss guides the training with all samples and gives weight to support vectors. Soft margin CE loss and hybrid CE loss can be extended to most classification tasks and offer a wide range of applications and great potential.

1. Introduction

1.1. Background and Significance

In the specialized domain of signal detection and classification, the subdiscipline of audio signal detection/classification is used to identify the signal types in audio recordings and estimate their onset and offset times (known as audio event detection). This technological paradigm unveils expansive applicative potential, such as audio surveillance [1,2], voice wakeup [3], speaker verification [4,5], environmental sound recognition [6,7], and bird call detection [8]. Audio signal detection/classification demonstrates significant versatility, with its techniques being applicable in areas such as spatial electromagnetic spectrum awareness [9], signal parameter estimation [10], signal analysis [11], and modulation recognition [12]. The expansive applicability and the technological versatility of this approach underscore its critical role in advancing the capabilities of contemporary signal detection and classification frameworks.

1.2. Current Status of the Problem

The most typical signal detection–classification method is the Gaussian Mixture model–hidden Markov model (GMM-HMM) method [13], in which each type of signal is modeled by a three-state hidden Markov model, and each state is modeled by a Gaussian mixture model. With the development of deep learning, an increasing number of signal detection–classification methods based on deep learning have been proposed, such as feedforward neural networks (FNNs) [14,15], convolutional neural networks (CNNs) [16,17], and recurrent neural networks (RNNs) [18]. In 2017, Cakir et al. proposed the convolutional recurrent neural network (CRNN) [19], which combines CNN and RNN, with the former extracting frequency invariance and the latter modeling temporal dependencies. As signals have different durations, many researchers have given weight to multi-scale information in recent years. Lu et al. proposed the multi-scale recurrent neural network (MS-RNN) [20], which uses the Mel-band energies feature of audio recordings at different time resolutions to capture both fine-grained and long-term dependencies. Zhang et al. proposed the multi-scale time-frequency attention (MTFA) method [21], which extracts features of different scales through convolutional operation and max-pooling. Transformer-based methods have been successfully used for sound event classification and weakly labeled sound event detection, including AST [22], AST-SED [23], CNN-Transformer [24], and SEDT [25,26]. Refs. [27,28] constructed Transformer/self-attention-based models for polyphonic sound event detection, but their results indicate that Transformer/self-attention-based methods do not exhibit significant advantages in polyphonic sound event detection. To accelerate the running speed and improve performance, in our earlier study [29], single-scale fully convolutional networks (SS-FCNs) and multi-scale fully convolutional networks (MS-FCNs) were proposed. The SS-FCN employs dilated convolution to encapsulate temporal context, offering more rapid execution compared to RNNs. However, its effectiveness in signal detection is limited to specific types, as it operates with a single temporal dependency length. Conversely, the MS-FCN integrates a feature fusion module that combines temporal dependencies of varying lengths, enhancing its ability to handle long short-term dependencies and, thus, improving the detection capabilities across a broader range of audio signals [29]. However, MS-FCN neglects fine-grained dependencies and intermediate-length temporal dependencies. To overcome the limitations of MS-FCN [29], in a subsequent manuscript, MSFF-Net [30], the dilated mixed convolution module, and the cascaded parallel module were proposed. The former merges dilated and standard convolutions to model temporal information, while the latter merges temporal dependencies of different lengths in a cascading and parallel manner to capture rich temporal information.

Although audio signal detection and classification have made great progress, there are still significant challenges that need to be addressed. This manuscript focuses on two of these challenges:

• The first challenge involves obtaining richer long short-term dependencies. As described in Refs. [21,31]; as audio signals (such as sound events) have different durations, multi-scale information is indispensable for signal detection. Although SS-FCN [29] demonstrated that dilated convolution is effective for modeling temporal context, a single dilated convolutional layer in SS-FCN can only capture features with a single length of temporal dependencies. While MS-FCN can capture multi-scale information by merging features output from the dilated convolution of different layers, MS-FCN can capture multi-scale information but is limited by the layer of dilated convolution. Similar to MS-FCN, MSFF-Net [30] can only capture and merge a limited number of temporal dependencies. Thus, a network that can obtain richer temporal context information and merge them to capture richer long short-term dependencies is required.
• The second challenge involves selecting effective samples (support vectors) and using them to guide the training. As audio signal detection–classification is considered a frame-wise prediction task, samples are defined according to the time frame (or time segment) of several tens of milliseconds. In a recording, sound events, such as gunshots and glass break, usually have extremely limited durations, resulting in background samples taking a dominant proportion. Thus, there is a high class imbalance between the foreground and background in audio signal detection and classification datasets. In addition, class imbalances exist among different kinds of audio signals, e.g., in the TUT-SED 2017 dataset, the duration of car events exceeds that of brakes squeaking, children, large vehicle, and people speaking events combined. Moreover, the dataset contains large silent clips in the background, which show high similarity and are easily classified. Moreover, some events, e.g., gunshot, glass break, car, and large vehicle events, have smaller sample sizes, high inter-sample similarity, and high concurrency, making them difficult to detect. As mentioned above, a serious class imbalance exists in datasets. Some events have large sample sizes and are easy to detect, contributing little to the training. Conversely, other events may have small sample sizes and are difficult to fit, requiring the network to focus more on them. Addressing how to manage these two types of samples and allocate different levels of attention during training to improve detection performance is a significant challenge.

1.3. Proposed Solutions and Contributions

In response to these challenges, this paper proposes the HDC-Inception module, soft margin cross-entropy loss, and hybrid cross-entropy loss (soft margin CE loss and hybrid CE loss) to capture richer long short-term dependencies and improve the utilization of effective samples.

• For the first challenge, this paper proposes the HDC-Inception module to capture richer long short-term dependencies, with the advantages of the Inception module [32,33] and hybrid dilated convolution (HDC) framework [34]. In the Inception module (as shown in Figure 1a), several parallel paths are constructed, and convolutional filters of different sizes are used in the parallel paths to capture and fuse features with different views. In our early SS-FCN [29], dilated convolution was used to model temporal context, as dilated convolution can ensure high time resolution and capture longer temporal dependencies without altering the filter size or network depth. Combining the Inception module and dilated convolution, the Dilated-Inception module (as shown in Figure 1b) is naturally proposed, which uses the architecture of the Inception module and replaces the convolutional filters of different sizes with dilated convolutions of different dilation factors. However, the Dilated-Inception module suffers from a “gridding issue” [34] caused by the element-skipping sampling of standard dilated convolution, under which some neighboring information is ignored [35]. Fortunately, the “gridding issue” can be alleviated by HDC [34], which uses a range of dilation factors and concatenates them serially to gradually increase the views. Based on the above analyses, we propose the HDC-Inception module (as shown in Figure 1c), which combines the advantages of the Inception module and HDC. The proposed HDC-Inception module has several parallel paths, which use dilated convolutions of different dilation factors to capture different temporal dependencies. Inspired by HDC, the dilation factors are increased path-by-path, and the output features of the previous path are sent to the next path to alleviate the “gridding issue”. Features from all paths are concatenated and fed into a convolutional filter to fuse information altogether. In addition, skip connections are used between two HDC-Inception modules to alleviate gradient vanishing. Obviously, by employing HDC-Inception modules in a stacked configuration, temporal dependencies of diverse durations can be acquired, thereby enabling networks to capture more comprehensive temporal contextual information.
• For the second challenge, this paper proposes soft margin cross-entropy loss (soft margin CE loss) to adaptively select effective samples (support vectors) and use them to guide the training, as inspired by the soft margin support vector machine (soft margin SVM) [36,37]. In SVM, when samples are inseparable in linear space, a non-linear kernel will be used to map the samples into a feature space, where a linear separating hyperplane can be found. As neural networks can be regarded as non-linear kernels of SVM, in the following discussion on SVM and loss function, samples are located in the space after non-linear mapping. In SVM (as shown in Figure 2a) [36,37], a hyperplane is found to separate the samples into two classes with the max margin. Based on the hyperplane, samples are divided into support vectors and non-support vectors. Only support vectors are used to optimize the separating hyperplane. However, SVM is only feasible when the samples are linearly separable. To overcome this limitation, the soft margin SVM (as shown in Figure 2b) is proposed, which allows misclassified samples and minimizes errors. Inspired by soft margin SVM, we propose soft margin CE loss (as shown in Figure 2c). Similar to soft margin SVM, the proposed soft margin CE loss presupposes a separating hyperplane and sets a margin to divide samples into support vectors and non-support vectors. Support vectors are used to calculate the optimum separating hyperplane. Through identity transforms, the calculation of this hyperplane is transformed into an optimization model/problem, which can be used as the loss function of neural networks. Although the proposed soft margin CE loss originated from soft margin SVM, which only works for binary classification problems, it applies to multi-classification tasks. Using the advantages of soft margin CE loss and CE loss, this paper also proposes a hybrid CE loss approach. This method utilizes all samples in training while focusing on support vectors.

In summary, this paper proposes the HDC-Inception module and soft margin CE loss to capture richer long short-term dependencies and improve the utilization of effective samples. The HDC-Inception module combines the advantages of the Inception module [32,33] and HDC [34], which can extract rich long short-term dependencies and alleviate the “gridding issue”. Soft margin CE loss couples the advantages of soft margin SVM and CE loss, which selects support vectors adaptively and uses them to guide the training. Combining soft margin CE loss and CE loss, this paper also proposes the hybrid CE loss approach, which uses all samples in the training and gives weight to support vectors. Soft margin CE loss and hybrid CE loss can be used as the loss functions of networks and are suitable for most classification tasks.

The rest of this paper is organized as follows: the related works are shown in Section 2. The proposed HDC-Inception module and soft margin CE loss are described in Section 3. Section 4 introduces the datasets and metrics. The results and discussion are presented in Section 5. Section 6 presents the conclusion and further works.

2. Related Works

2.1. Audio Signal Detection and Classification

Early signal detection and classification methods included the Gaussian mixture model–hidden Markov model (GMM-HMM) method [13] and the non-negative matrix factorization (NMF) method [38]. As the technology evolved, deep learning methods emerged, leading to the adoption of feedforward neural networks (FNNs) [14,15] and convolutional neural networks (CNNs) [16,17] in this field. Limited by time resolution, FNN [14,15] and CNN [16,17] are not suitable for signal detection and classification. To overcome this limitation, Parascandolo et al. proposed the RNN-based method [18], which uses RNNs [39] to predict the class of each signal frame by frame, ensuring high time resolution. Furthermore, by combining RNNs (such as long short-term memory (LSTM) [40]) with hidden Markov models, Hayashi et al. proposed the duration-controlled long short-term memory [41], which utilizes three-state left-to-right hidden Markov models to describe the course of audio events. Using the advantages of CNNs [16] and RNNs [18], Cakir et al. proposed well-known convolutional recurrent neural networks (CRNNs) [19] for audio signal detection and classification, where CNNs are used to extract frequency invariance and RNNs ensure high time resolution. Despite the attractive performance of CRNNs, which has led to expanded versions such as C3RNN [42], R-CRNN [43], and [44,45], these methods have seen limited performance improvements. Although Transformer-based approaches have been successful in sound event classification and weakly labeled sound event detection [22,23,24,25,26,46,47,48,49], for polyphonic sound event detection, these methods do not offer substantial benefits [27,28]. In recent years, the concept of long short-term dependencies has garnered attention from researchers. Lu et al. proposed multi-scale recurrent neural networks (MS-RNNs) [20], in which audio recordings of different time resolutions are used to capture temporal dependencies of different lengths. Zhang et al. proposed the multi-scale time-frequency attention (MTFA) method [21], employing convolutional operations and max-pooling to capture multi-scale features.

However, all these methods mentioned above use RNNs (e.g., LSTM [40] or gated recurrent unit (GRU) [50]) to model temporal context information; therefore, they share the RNN limitations of training challenges and low running speeds To overcome these challenges, the SS-FCN and MS-FCN approaches are proposed [29]. SS-FCN demonstrates that dilated convolution can be used to model temporal context information [29]. MS-FCN demonstrates that fusing temporal dependencies of different lengths can enhance the performances of different kinds of signals. As an improvement of MS-FCN, MSFF-Net [30] is capable of capturing richer temporal dependencies. SS-FCN, MS-FCN, and MSFF-Net approaches achieve competitive performance and their running speeds are well above those of previous methods [29,30].

2.2. Inception Module and Hybrid Dilated Convolution

Inception module: The Inception module (as shown in Figure 1a) was first proposed by Szegedy et al. for object detection and classification [33]; it arranges parallel convolutions of different kernels to capture different views from the input feature. As the Inception module can capture features with different views and deal with multi-scale problems of objects, it has been extensively applied to object recognition [51], human pose estimation [52], action recognition [53,54,55], and so on.

Hybrid dilated convolution: Hybrid dilated convolution (HDC) [34] was proposed by Wang et al. to alleviate the “gridding issue” caused by dilated convolution. In dilated convolution, zeros are padded between two pixels in a convolutional kernel; thus, the receptive field of this kernel only covers an area with checkerboard patterns, and some neighboring information is ignored. To alleviate the “gridding issue”, HDC uses a range of dilated convolutions with different dilation factors and concatenates them in series, under which the receptive field grows gradually and covers the whole area.

2.3. Soft Margin Support Vector Machine

Support vector machine (SVM), proposed by Cortes et al. [36], is a classical machine learning algorithm and has been extensively applied to classification tasks. As shown in Figure 2a, SVM calculates a separating hyperplane with the max-margin to separate the samples into two classes. Samples, closest to the hyperplane are defined as support vectors. In SVM, only support vectors are used to calculate the optimum hyperplane, and non-support vectors do not contribute to the calculation of the hyperplane. The training sample is denoted by $x_i\in {\mathbb{R}}^p$ with label $y_i\in \{+1,-1\}$,$i=\{1,2,\dots ,n\}$. As shown in Figure 2a, the optimum hyperplane $wx+b=0$ separates two classes with the largest margin and all samples meet the constraint of $y(wx+b)\ge 1$, where $w\in {\mathbb{R}}^p$ represents the weight vector of the separating hyperplane and b is the bias. To maximize the margin, the optimum separating hyperplane can be obtained by solving the optimization problem in Equation (1):

$$\begin{array}{cc}\hfill & \min {\displaystyle \frac{1}{2}}{∥w∥}^2\hfill \\ \hfill & \mathrm{subject}\mathrm{to}{y}_i(w{x}_i+b)\ge 1,i=1,\dots n.\hfill \end{array}$$

(1)

However, SVM requires that all samples follow the constraint of $y(wx+b)\ge 1$, causing SVM to be feasible only when the samples are separable linearly. To overcome this limitation, the soft margin SVM is proposed, which allows misclassified samples that do not meet the constraint, as shown in Figure 2b. In soft margin SVM, the misclassified samples incur the error $1-y(wx+b)$. Taking this error as a penalty and integrating it into Equation (1), the optimization problem transforms into Equation (2):

$$\begin{array}{c}\hfill \underset{w,b}{\min }{\displaystyle \frac{1}{2}}{∥w∥}^2+C\sum _{i=1}^{n}max(0,1-y_i(w{x}_i+b))\end{array}$$

(2)

where C is a regularization parameter that defines the trade-off between the maximum margin and the misclassified error.

3. Method

3.1. HDC-Inception

The proposed HDC-Inception approach takes advantage of both the HDC [34] and Inception modules [32,33], with the former alleviating the “gridding issue” and the latter capturing temporal dependencies of different lengths. Ref. [29] demonstrated that dilated convolution is effective at modeling temporal context information in audio signal detection and classification systems. Dilated convolution can address the limitations of RNNs, including restricted temporal dependencies, training difficulties, and low efficiency. In the HDC-Inception model, dilated convolution is specifically chosen to model temporal context.

As shown in Figure 1a, in video tasks, through constructing several parallel paths with convolutional filters of different sizes, the Inception module can capture features of different scales. Extending to signal processing, the Dilated-Inception module is proposed, as shown in Figure 1b, in which parallel paths with different dilation factors are used to capture temporal dependencies of different lengths. However, dilated convolution suffers from the “gridding issue”. To overcome this limitation, inspired by HDC [34], HDC-Inception is proposed (Figure 1c). This approach gradually increases the dilation factor in the parallel path; the output of the previous path, concatenated with the input, is fed into the next path to ensure the gradual increase in the temporal dependent length. Obviously, in HDC-Inception, different paths output temporal dependencies of different lengths. To better fuse temporal dependencies of different lengths, the outputs of all paths and the input features are concatenated and then passed through a convolution with a filter size of 1. In addition, skip connections are used between two HDC-Inception modules to alleviate gradient vanishing. Referring to [56], the exponential dilation factors of {1, 2, 4, 8, 16,…} are selected in HDC-Inception. As too-large dilation factors lead to too-sparse sampling, failing to cover local temporal context information, the proposed HDC-Inception module utilizes four parallel paths with dilation factors of {1, 2, 4, 8}. Hence, one HDC-Inception obtains temporal dependencies of four different lengths. As HDC-Inception modules are connected in series, if there are $N_{HI}$ HDC-Inception modules in the proposed method, $4^{N_{HI}}$ temporal dependencies are captured. Thus, through HDC-Inception, the proposed method can capture rich temporal contextual information.

3.2. Soft Margin CE Loss

Inspired by soft margin SVM, the proposed soft margin CE loss can identify support vectors and non-support vectors. Furthermore, soft margin CE loss meets the general requirements of classification tasks and can be used for multi-label and multi-classification tasks. In classification tasks, classification methods based on neural networks usually output a prediction, ranging from 0 to 1, to represent the activity probability of one class. In this process, the neural networks can be regarded as non-linear kernels of SVM, which map the samples into a feature space, where samples satisfy linear separability. Based on this, as shown in Figure 2c, the activity probability for the sample is in the range of [0, 1] ($p\in [0,1]$). Analogous to SVM, we assume that the separating hyperplane is $p=0.5$, which is used to judge whether an event occurred.

When the sound event occurs in this audio segment (the target output y = 1), the constraint is shown in Equation (3), as follows:

$$\begin{array}{c}\hfill p\ge 0.5+E\end{array}$$

(3)

where E represents the margin. Via an identity transformation, this constraint converts to Equation (4), as follows:

$$\begin{array}{c}\hfill -log\left(p\right)\le -log(0.5+E)\end{array}$$

(4)

In the proposed soft margin CE loss, the misclassified samples are allowed, which do not meet the constraint equation, Equation (3). Similar to soft margin SVM, the misclassified samples incur the error $-log\left(p\right)+log(0.5+E)$ and are regarded as support vectors. Taking this error as a penalty and considering all samples, the optimization function can be formulated as in Equation (5):

$$\begin{array}{c}\hfill min[max(0,-log\left(p\right)+log(0.5+E)\left)\right]\end{array}$$

(5)

When there is no sound event in this audio segment (target output y = 0), the constraint can be denoted as in Equation (6):

$$\begin{array}{c}\mathrm{p}\le 0.5-E\hfill \\ \Rightarrow 1-p\ge 0.5+E\hfill \\ \Rightarrow -log(1-p)\le -log(0.5+E)\hfill \end{array}$$

(6)

In this condition, the misclassified samples incur the error $-log(1-p)+log(0.5+E)$ and the optimization function is shown in Equation (7), as follows:

$$\begin{array}{c}\hfill min(max(0,-log(1-p)+log(0.5+E)\left)\right)\end{array}$$

(7)

Considering all situations and all samples, we can obtain the soft margin CE loss, as shown in Equation (8):

$$\begin{array}{c}\hfill min[\sum _{i=1}^n\begin{array}{c}(y_i\ast max(0,-log\left(p_i\right)+log(0.5+E))+\hfill \\ (1-y_i)\ast max(0,-log(1-p_i)+log(0.5+E))\hfill \end{array}]\end{array}$$

(8)

where $y_i=\{0,1\}$ is the target output and $E\in [0,0.5]$ represents the margin.

Analysis: In this section, we analyze the characteristics of the proposed soft margin CE loss and compare them with CE loss. In this analysis, a simple classification model is selected, which consists of a single-layer fully connected network followed by a sigmoid. In this model, the forward propagation is shown as follows:

$$\begin{array}{c}\hfill s=w\odot x=\sum _{j=1}^J{w}_j\ast x_j\end{array}$$

(9)

$$\begin{array}{c}\hfill p={\displaystyle \frac{1}{1+e^{-s}}}\end{array}$$

(10)

$$\begin{array}{c}\hfill L_{CE}=\sum _{i=1}^N(-y_i\ast log\left(p_i\right)-(1-y_i)\ast log(1-p_i))\end{array}$$

(11)

$$\begin{array}{cc}\hfill & L_{SM-CE}=\sum _{i=1}^N(y_i\ast max(0,-log\left(p_i\right)+log(0.5+E))\hfill \\ \hfill & +(1-y_i)\ast max(0,-log(1-p_i)+log(0.5+E))\hfill \end{array}$$

(12)

where s is the output of the fully connected network, and p is the output of the sigmoid representing the activity probability of a sound event. $L_{CE}$ is the CE loss and $L_{SM-CE}$ is the soft margin CE loss. ⊙ denotes the inner product. The purpose is to minimize $L_{CE}$ or $L_{SM-CE}$. The partial derivative ${\displaystyle \frac{\partial L}{\partial w_j}}$ represents the speed of convergence, where $L\in \{L_{CE},L_{SM-CE}\}$.

When CE loss is selected as the loss function, the calculation of the partial derivative ${\displaystyle \frac{\partial L_{CE}}{\partial w_j}}$ is shown as follows:

$$\begin{array}{cc}\hfill & \partial L_{CE}/\partial p=-y/p+(1-y)/(1-p)\hfill \\ \hfill & \partial p/\partial s=(1/(1+e^{-s}))\ast (e^{-s}/(1+e^{-s})=p\ast (1-p)\hfill \\ \hfill & \partial s/\partial w_j=x_j\hfill \\ \hfill & so:\hfill \\ \hfill & {\displaystyle \frac{\partial L_{CE}}{\partial w_j}}={\displaystyle \frac{\partial L_{CE}}{\partial p}}{\displaystyle \frac{\partial p}{\partial s}}{\displaystyle \frac{\partial s}{\partial x_j}}\hfill \\ \hfill & =[-y/p+(1-y)/(1-p)]\ast p\ast (1-p)\ast x_j\hfill \\ \hfill & =[-y\ast (1-p)+(1-y)\ast p]\ast x_j\hfill \\ \hfill & =(p-y)\ast x_j\hfill \end{array}$$

(13)

Referring to Equation (13), when soft margin CE loss is selected as the loss function, the computational formula of the partial derivative ${\displaystyle \frac{\partial L_{SM-CE}}{\partial w_j}}$ is as follows:

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial L_{SM-CE}}{\partial w_j}}=\left\{\begin{array}{cc}0,& (y=0\mathrm{and}p\le 0.5-E)\\ & \mathrm{or}(y=1\mathrm{and}p\ge 0.5+E)\\ (p-y)\ast x_j,& others\end{array}\right.\end{array}$$

(14)

For CE loss and the proposed soft margin CE loss, the comparisons between loss values and their corresponding partial derivatives ${\displaystyle \frac{\partial L}{\partial w_j}},L\in \{L_{CE},L_{SM-CE}\}$ are shown in Figure 3. Observing Figure 3, the proposed soft margin CE loss has similar characteristics to CE loss; for this reason, soft margin CE loss can replace CE loss in most classification tasks. The difference is that CE loss uses all samples to train the networks, but soft margin CE loss selects support vectors and uses them to guide the training, in which non-support vectors do not contribute to training.

3.3. Hybrid CE Loss

Although soft margin CE loss can select support vectors, which play an indispensable role in training, the number of support vectors is limited. However, neural networks need huge samples to support training. Under comprehensive consideration, the hybrid CE loss function, mixing the proposed soft margin CE loss with classical CE loss, is proposed, as shown in Equation (15).

$$\begin{array}{c}\hfill L=r\ast L_{SM-CE}+(1-r)\ast L_{CE}\end{array}$$

(15)

where $L_{SM-CE}$ denotes soft margin CE loss, $L_{CE}$ denotes classical CE loss, and L denotes hybrid loss. r denotes the trade-off between soft margin CE loss and classical CE loss.

3.4. Proposed Method

Employing HDC-Inception, along with soft margin CE loss and hybrid CE loss, forms the core of the proposed methodology. As shown in Figure 4, the proposed method consists of three parts: (1) Convolutional layers used to obtain features with frequency invariance, (2) HDC-Inception layers, which can model temporal context and capture long short-term dependencies, and (3) loss functions, under which CE loss, soft margin CE loss, and a hybrid loss function that combines both are selected. Next, we will go into detail about each of the three parts.

3.4.1. Preprocessing

In this paper, we select the log Mel-band energy (MBE) as input for the neural networks, which has been extensively used for audio signal detection and classification [19,29]. Firstly, to increase comparability with previous methods, each audio recording is divided into 46.4 ms segments (or frames) with 50% overlap using the hamming window, similar to [19,20,29,30,42]. Then, the 40-band MBE $x_t\in R^F$ is extracted from each audio signal segment, considering the computational load and comparability with previous methods [19,20,29,30,42], where $F=40$ denotes the number of features. Referring to [19,29], we select 256 consecutive MBEs (e.g., $X=\{{\mathrm{x}}_0,{\mathrm{x}}_1\dots {\mathrm{x}}_{T-1}\}\in {\mathbb{R}}^{F\times T},T=256$) as the input for the networks. The target output corresponding to the input X is denoted by $Y=\{y_0,y_1\dots y_{T-1}\}\in {\mathbb{R}}^{N_c\times T}$, where $N_c$ indicates the type of target event; $y_t={\{y_{(0,t)},y_{(1,t)}\dots y_{(N_c-1,t)}\}}^T\in {\mathbb{R}}^{N_c}$ represents the label of the t-th frame, and if the c-th class occurs in the t-th frame, $y_{(c,t)}$ equals 1, otherwise, 0.

3.4.2. Convolutional Layers

The MBE sequence X is fed into the convolutional layers, in which two-dimensional convolutional filters are used and each convolution is followed by batch normalization, ReLU activation, and non-overlapped max-pooling. As the convolutional layers are used to enhance the frequency invariance, the max-pooling is computed only along the frequency axis and the time resolution of features remains the same. The output of convolutional layers is indicated by $F_c\in {\mathbb{R}}^{M\times \overline{F}\times T}$, where M represents the number of channels (or the hidden units) and $\overline{F}$ denotes the number of frequency bands.

3.4.3. HDC-Inception Layers

The HDC-Inception layers are composed of multiple HDC-Inception modules; each layer is followed by batch normalization and ReLU activation. The outputs of the convolutional layers are stacked along the frequency axis and then fed into the HDC-Inception layers. The output of the $n-th$ Dilated-Res module is represented by $F_{HI}^n\in {\mathbb{R}}^{W^n\times T}$. And $n\le N_{HI},N_{HI}$ denotes the number of HDC-Inception modules in the proposed method.

3.4.4. Prediction Layer

The output of HDC-Inception layers $F_{HI}^{N_{HI}}$ is fed into the prediction layer to predict the activity probabilities of sound events. The prediction layer is a single convolutional layer with a filter size of 3 and sigmoid activation. The output is denoted by $P\in {\mathbb{R}}^{N_c\times T}$, which is a T-length sequence, in which $P(c,t)\in (0,1)$ denotes the activity probability of class c in the $t-th$ audio frame.

3.4.5. Loss Function

The proposed soft margin CE loss and hybrid CE loss are selected as the loss functions in the proposed method. Soft margin CE loss gives weight to difficult samples (renamed as support vectors) and hybrid CE loss takes advantage of soft margin CE loss and classical CE loss; this approach strikes a balance between considering all samples and focusing on the more difficult samples.

Batch normalization is used after each convolutional or dilated convolutional layer to reduce the internal covariate shift. And L2 regularization is selected to prevent overfitting. Adam [57] is set as the gradient descent optimizer.

4. Experiments

4.1. Dataset

We test the proposed method on three datasets: the TUT Rare Sound Events 2017 dataset, the TUT-SED 2017 dataset, and the TUT-SED 2016 dataset. (1) The TUT Rare Sound Events 2017 dataset [58] is a synthetic dataset, with three types of isolated audio signals as the target events and backgrounds coming from 15 different scenes. The target events include ’baby cry’ (106 training, 42 test instances, mean duration 2.25 s), ‘glass break’ (96 training, 43 test instances, mean duration 1.16 s), and ‘gunshot’ (134 training, 53 test instances, mean duration 1.32 s). For the development training dataset, 5000 mixtures were generated per target event, incorporating event-to-background ratios of −6, 0, and 6 dB, alongside an event presence probability of 0.99. In contrast, the development testing set and the evaluation set utilize the official set of 500 audio samples per target event provided for testing. (2) The TUT-SED 2017 dataset [58] comprises recordings from real-life street scenes featuring six sound event classes. The development dataset includes 24 files, totaling 70 min. The dataset is divided into training and testing sets using the official four-fold cross-validation method for experiments. In this dataset, the number of instances for each event class includes ‘brakes squeaking’ (52 development, 24 evaluation), ‘car’ (304 development, 110 evaluation), ‘children’ (44 development, 19 evaluation), ‘large vehicle’ (61 development, 24 evaluation), ‘people speaking’ (89 development, 47 evaluation), and ‘people walking’ (109 development, 48 evaluation) events. (3) Recording from two real-life scenes, the TUT-SED 2016 dataset [59] consists of 22 recordings ranging from three to five minutes each. Of these, 10 recordings are from homes and feature 11 sound events, while 12 recordings are from residential areas with 7 sound events. Details on the number of instances per class are provided in

Table 1. Event instances per class in the TUT Rare Sound Events 2017 dataset, the TUT-SED 2017 dataset, and the TUT-SED 2016 dataset.

TUT Rare Sound Events 2017 Dataset			TUT-SED 2017 Dataset			TUT-SED 2016 Dataset
Event	Development	Evaluation	Event	Development	Evaluation	Event (Home)		Event (Residential Area)
baby cry	106	42	brakes squeaking	52/96.99 s	24	(object) Rustling	60	(object) Banging	23
glass break	96	43	car	304/2471.14 s	110	(object) Snapping	57	Bird singing	271
gunshot	134	53	children	44/35.99 s	19	Cupboard	40	Car passing by	108
			large vehicle	61/923.74 s	24	Cutlery	76	Children shouting	31
			people speaking	89/715.66 s	47	Dishes	151	People speaking	52
			people walking	109/1246.95 s	48	Drawer	51	People walking	44
						Glass jingling	36	Wind blowing	30
						Object impact	250
						People walking	54
						Washing dishes	84
						Water tap running	47

of [59].

For these three datasets, the per-class event instances can be found in Table 1. One instance represents a complete process of one sound event, with a duration ranging from a few tenths of a second to several tens of seconds. All three datasets are publicly available datasets and more detailed information about them can be found in Refs. [58,59].

4.2. Metrics

Building on previous methods [19,29,30], event-based metrics [60] assess the TUT Rare Sound Events 2017 dataset by comparing system outputs to the corresponding reference outputs on an event-by-event basis. Meanwhile, segment-based metrics [60] are applied to evaluate the TUT-SED 2016/2017 datasets, analyzing performances over continuous segments, while segment-based metrics compare the system output and reference (the target output) in short time segments [60]. Both employ intermediate statistics (true positive, false positive, and false negative), which are used to calculate the F1 score and error rate (ER). More details about these metrics can be found in Ref. [60].

4.3. Baseline

Our earlier proposed SS-FCN [29] is selected as the baseline. The proposed method has a framework similar to SS-FCN, which is composed of convolutional layers, dilated convolutional layers, and CE loss. Convolutional layers in SS-FCN and the proposed method are the same. The dilated convolutional layers of SS-FCN and CE loss correspond to the proposed HDC-Inception layers and soft margin CE loss (or hybrid loss), respectively.

4.4. Experiments Setup

Referring to [19,29,30], for each dataset, the Mel-band energy sequences used as input are consistently set to a length of 256, except for the evaluation dataset of the TUT Rare Sound Events 2017, where the entire file with an MBE sequence length of 1293 is used. The convolutional layers are 2D-CNNs with a filter size of 3; according to Refs. [19,29], the hyperparameter grid search is launched on the hidden units of {32, 64, 128} and the frequency max-pooling arrangements {(5, 4, 2), (5, 2, 2)}. The proposed HDC-Inception module has five parallel paths, in which four are dilated convolutions with dilation factors {1, 2, 4, 8} and one is a skipping connection. The HDC-Inception model undergoes a hyperparameter optimization using a grid search across the values {32, 64, 128} for its hidden units. A convolutional network with a filter size of 3 and a sigmoid activation function is chosen for the prediction layer. For the proposed soft margin CE loss, we run a hyperparameter grid search on the margin E {0.1, 0.2, 0.3, 0.4}. For the hybrid loss function, we run a hyperparameter grid search on the trade-off coefficients r {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}. As the separating hyperplane is $p=0.5$ in the soft margin CE loss approach, the event activity probabilities P are thresholded at 0.5. The starting learning rate is established at 0.001. Concerning the TUT Rare Sound Events dataset, the training is capped at 50 epochs, implementing early stopping if there is no decrease in validation loss for ten consecutive epochs. Concerning the TUT-SED 2017/2016 dataset, the training is extended to a maximum of 1000 epochs, with early stopping triggered if there is no reduction in the overall error rate on the validation set after 100 epochs. Each experiment is conducted five times using the official four-fold cross-validation setup to obtain average results.

5. Results and Discussion

The experiments are essentially split into two segments. Initially, ablation experiments are performed on the TUT Rare Sound Events 2017 dataset [58] to validate the efficiency of HDC-Inception, soft margin CE loss, and hybrid CE loss. Subsequently, comparisons with existing methods are conducted on the TUT Rare Sound Events 2017 dataset [58], TUT-SED 2017 dataset [58], and TUT-SED 2016 dataset [59]. The experimental results show that the HDC-Inception and hybrid CE loss proposed in this paper can effectively enhance performance and outperform existing audio signal detection and classification methods across several datasets.

5.1. Ablation Studies

This section prioritizes the validity of HDC-Inception, followed by the performances of soft margin CE loss and hybrid CE loss. It was observed that the performance of the method based on soft margin CE loss is limited. The reason is that under soft margin CE loss, the number of support vectors in a batch is too small, making it difficult to improve the performance. To tackle this problem, this paper proposes hybrid CE loss, and experiments show that the hybrid CE loss can effectively improve the performance of the method.

5.1.1. HDC-Inception Module

Effect of HDC-Inception In this section, SS-FCN [29] is chosen as the baseline, which consists of convolutional layers, dilated convolutional layers, and CE loss. The proposed HDC-Inception layer is used to replace dilated convolution layers, in the SS-FCN, to obtain the method named SS-FCN + HDC-Inception. To verify the effectiveness of the HDC-Inception proposed, this section compares the performance of SS-FCN and SS-FCN+HDC-Inception. In this experiment, the number of dilated convolution layers and HDC-Inception layers is the same, consisting of 4, 5, 6, and 7, respectively. For the sake of fairness, the results of SS-FCN in Refs. [29,30] are selected. Compared to SS-FCN+HDC-Inception, the results are shown in

Table 2. HDC-Inception vs. dilated convolution. Layers: the number of dilated convolution layers or HDC-Inception layers. ER: class-wise averaging error rate, F1: class-wise averaging F1 score, development|evaluation: TUT Rare Sound Event 2017 development/evaluation dataset. The best results are marked in bold face.

	SS-FCN		SS-FCN+HDC-Inception
Layer	Development|Evaluation		Development|Evaluation
	ER	F1(%)	ER	F1(%)
4	0.15|0.27	92.03|86.74	0.12|0.18	93.84|90.99
5	0.17|0.20	91.59|90.16	0.10|0.16	94.38|91.43
6	0.12|0.17	94.12|91.47	0.11|0.16	94.52|91.87
7	0.10|0.18	94.81|90.60	0.11|0.17	94.12|91.58

. Four networks with two metrics, plus the development dataset and the evaluation dataset, make a total of 16 comparable indicators. As shown in Table 2, SS-FCN+HDC-Inception outperforms SS-FCN in 14 of the 16 metrics. Thus, the proposed HDC-Inception approach has a better ability to model temporal contextual information compared to dilated convolution.

The layer of HDC-Inception. This section analyzes the impact of the number of HDC-Inception layers on detection performance. The results are displayed in

Table 3. Effects of the layer of HDC-Inception. Networks: SS-FCN+HDC-Inception. Layer: the layer of HDC-Inception. ER: error rate. F1: F1 score. The best results are marked in bold face.

	Development Dataset				Evaluation Dataset
Layer	Baby Cry	Glass Break	Gunshot	Average	Baby Cry	Glass Break	Gunshot	Average
	ER|F1	ER|F1	ER|F1	ER|F1	ER|F1	ER|F1	ER|F1	ER|F1
4	0.09|95.39	0.05|97.34	0.21|88.79	0.12|93.84	0.28|86.91	0.14|92.34	0.12|93.71	0.18|90.99
5	0.07|96.54	0.04|97.97	0.20|88.64	0.10|94.38	0.15|92.40	0.16|91.07	0.17|90.83	0.16|91.43
6	0.13|93.44	0.06|97.14	0.14|92.98	0.11|94.52	0.16|92.25	0.17|91.10	0.16|92.25	0.16|91.87
7	0.11|94.80	0.06|96.69	0.17|90.87	0.11|94.12	0.28|86.91	0.14|92.47	0.09|95.37	0.17|91.58
8	0.09|95.51	0.07|96.52	0.21|88.40	0.12|93.48	0.23|88.49	0.15|91.95	0.15|92.21	0.18|90.88
9	0.14|93.15	0.06|97.13	0.21|88.60	0.14|92.96	0.17|91.62	0.17|91.18	0.15|92.43	0.16|91.74

, with the best performance in bold. As shown in Table 3, the best performance is achieved with 4 to 7 HDC-Inception layers. As the number of HDC-Inception layers increases from 4 to 6, the class-wise average error rate tends to decrease, and the F1 score tends to improve on both the evaluation and development datasets. As the number of layers increases from 6 to 8, both the class-wise average error rate and the F1 score deteriorate on the evaluation dataset, and the performance on the development dataset also worsens. Therefore, in the task of audio signal detection and classification, more HDC-Inception layers do not necessarily result in better performance. Although the network with nine HDC-Inception layers performs well on the evaluation dataset, its performance is poor on the development dataset. With six HDC-Inception layers, the network achieves the best class-wise average F1 score and the lowest class-wise average error rate on both the development and evaluation datasets. Thus, the optimal number of HDC-Inception layers is set to 6.

5.1.2. Soft-Margin CE Loss

Effect of margins (E). This section is used to verify the validity of the soft margin CE loss and the effect of margins (E) in this loss function; the results are shown in

Table 4. Effect of margin in soft margin CE loss. Dataset: TUT Rare Sound Events 2017 dataset. Margin (E): the margin of the soft margin CE loss. The best results are marked in bold face.

	Development Dataset				Evaluation Dataset
Margin (E)	Baby Cry	Glass Break	Gunshot	Average	Baby Cry	Glass Break	Gunshot	Average
	ER|F1	ER|F1	ER|F1	ER|F1	ER|F1	ER|F1	ER|F1	ER|F1
0.1	0.16|91.87	0.10|95.00	0.18|90.11	0.15|92.33	0.25|87.80	0.13|92.93	0.12|93.70	0.17|91.48
0.2	0.15|92.71	0.06|96.69	0.24|87.01	0.15|92.14	0.17|91.55	0.16|91.18	0.14|92.89	0.16|91.87
0.3	0.17|91.55	0.10|94.51	0.18|90.64	0.15|92.23	0.21|89.47	0.15|92.01	0.11|94.50	0.15|91.99
0.4	0.14|93.03	0.10|94.74	0.18|89.87	0.14|92.55	0.19|90.47	0.14|92.27	0.16|91.45	0.16|91.40
0.45	0.13|93.76	0.05|97.56	0.26|85.90	0.15|92.41	0.20|90.02	0.14|92.89	0.16|91.96	0.17|91.62

and Figure 5. For the baby cry and glass break events, the best error rates and F1 scores are obtained by soft margin CE loss with $E=0.45$ on the development dataset. However, on the evaluation dataset, the best performances are obtained at $E=0.2$ and $E=0.1$, respectively. For gunshot, on the development and evaluation datasets, when $E=0.3$, the best performance is attained. Observing the best performance for each event in Table 4 and Figure 5, the best performances for the development dataset are distributed at E equal to {0.3, 0.4, 0.45}, and for the evaluation dataset, the best performances are distributed at E equal to {0.1, 0.2, 0.3}. As the results show, with a larger E, soft margin CE loss performs better on the development dataset, and with a smaller E, it performs poorly on the development dataset and relatively well on the evaluation dataset. The larger the E, the better the fit between the soft margin CE loss and the traditional CE loss. When $E=0.5$, soft margin CE loss is equal to CE loss. With large E, soft margin CE loss performs better on the development dataset and worse on the evaluation dataset. While with a small E, the performance on the development dataset degrades, there is better performance on the evaluation dataset. This result, to some extent, indicates that the soft margin CE loss proposed in this paper is somewhat resistant to overfitting. With $E=0.3$, the soft margin CE loss shows better performance on both the development and evaluation datasets; therefore, $E=0.3$ was chosen for subsequent experiments.

Effect of soft margin CE loss. As shown in

Table 5. Results of ablation studies. Dataset: TUT Rare Sound Events 2017 dataset. SM-CE: soft margin CE loss. The best results are marked in bold face.

	Development Dataset				Evaluation Dataset
	Baby Cry	Glass Break	Gunshot	Average	Baby Cry	Glass Break	Gunshot	Average
Method	ER|F1	ER|F1	ER|F1	ER|F1	ER|F1	ER|F1	ER|F1	ER|F1
SS-FCN	0.11|94.67	0.10|94.65	0.14|93.03	0.12|94.12	0.26|86.96	0.15|92.08	0.09|95.37	0.17|91.47
+HDC-Inception	0.13|93.44	0.06|97.14	0.14|92.98	0.11|94.52	0.16|92.25	0.17|91.10	0.16|92.25	0.16|91.87
+HDC-Inception+SM-CE	0.17|91.55	0.10|94.51	0.18|90.64	0.15|92.23	0.21|89.47	0.15|92.01	0.11|94.50	0.15|91.99
+HDC-Inception+hybrid-CE	0.14|93.09	0.07|96.54	0.14|92.66	0.12|94.10	0.20|90.41	0.10|94.61	0.10|95.12	0.13|93.38
MS-FCN	0.06|96.76	0.09|95.40	0.14|92.86	0.10|95.01	0.18|90.98	0.18|89.91	0.10|95.10	0.15|92.00
+SM-CE	0.15|92.56	0.10|95.00	0.13|93.19	0.12|93.58	0.17|91.53	0.16|91.30	0.12|93.50	0.15|92.11
+hybrid CE loss	0.15|92.77	0.04|97.97	0.15|92.01	0.11|94.25	0.20|90.20	0.14|92.74	0.10|95.10	0.14|92.68

, compared with SS-FCN+HDC-Inception, after soft margin CE loss is used, the performance improves slightly on the evaluation dataset and decreases significantly on the development dataset. To verify whether this drop is a coincidence, the comparative experiments based on MS-FCN are launched, in which the CE loss and soft margin CE loss are selected as the loss functions, respectively; the results are shown in the last two rows of Table 5. The results are similar to those of SS-FCN, with limited performance improvement on the evaluation dataset and more significant performance degradation on the development dataset after using soft margin CE loss instead of CE loss.

The difference between soft margin CE loss and CE loss causes this degradation. Observing Figure 3, CE loss can be regarded as a special soft margin CE loss with $E=0.5$. So, the differences between soft margin CE loss and CE loss are their different points of convergence. CE loss converges to 1 or 0, and soft margin CE loss only requires samples to meet $p\ge 0.5+E$ or $p\le 0.5-E$. In soft margin CE loss, when the output p of a positive sample satisfies $p\ge 0.5+E$, or p of a negative sample meets $p\le 0.5-E$, the partial derivative ${\displaystyle \frac{\partial L}{\partial w}}$ is equal to 0. Therefore, this kind of sample is not valid for training and is called the non-support vector. It is known that training a neural network requires a large dataset. However, when soft margin CE loss is used, although the training of support vectors is well noticed, this loss function can cause many samples to become non-support vectors prematurely, which is detrimental to the training. In the task of audio signal detection–classification, the input is a continuous sequence of Mel spectra, represented as $X=\{x_0,x_1\dots x_{T-1}\}\in {\mathbb{R}}^{F\times T}$, where T = 256 denotes the sequence length. In this experiment, this X will be noted as a support vector, if one element $x_t$ of X is a support vector. Figure 6 presents the number of support vectors and the ratio in the dataset after each epoch. Observing Figure 6, the proportion of support vectors is about 25% in the later epochs. In this experiment, the batch size is 256, and considering the support vectors, the effective batch size is about 256∗0.25 = 64. Note that not all elements work in a support vector (X); thus, the actual number of effective samples is extremely low. To address this problem, this paper proposes the hybrid CE loss, which can use all samples to train the network while focusing on the support vector.

5.1.3. Hybrid CE Loss

The hybrid CE loss can be trained using all samples while focusing on the support vectors. The experimental results are shown in the fourth and last row of Table 5. As shown in Table 5, when SS-FCN is selected as the baseline and hybrid CE loss is used to replace soft margin CE loss, for the development dataset, the class-wise average error rate decreases by 3% and the class-wise average F1 score improves by 1.87%. For the evaluation dataset, the corresponding improvements are 2% and 1.39%. Comparing MS-FCN with soft margin CE loss, after hybrid CE loss is selected, the class-wise average error rate improves by 1%, and the class-wise F1 score improves by 0.67%. Compared to MS-FCN with traditional CE loss, hybrid CE loss is less effective for the development dataset, while for the evaluation dataset, improvements of 1% and 0.68% are obtained for the class-wise error rate and F1 score. In summary, hybrid CE loss provides overall improvement compared to soft margin CE loss and shows greater robustness than traditional CE loss.

5.2. Comparisons with Existing Methods

In this section, the hybrid CE loss is selected as the loss function for SS-FCN+HDC-Inception and MS-FCN to compare with existing methods. The experiment encompasses the TUT Rare Sound Event 2017 dataset, the TUT-SED 2017 dataset, and the TUT-SED 2016 dataset.

5.2.1. TUT Rare Sound Event 2017 Dataset

In ablation studies, for the TUT Rare Sound Event 2017 dataset, all sound events, such as baby cry, glass break, and gunshot, share a single network. While, in previous methods, a separate network was launched for each kind of sound event to improve the performance. To be fair, this section will construct dedicated networks for each type of sound event, and the results are shown in

Table 6. The results on the TUT Rare Sound Event 2017 dataset. The symbol “***” and “*******” indicates that specific results are not available. HDCi: HDC-Inception. The best results are marked in bold face.

	Development Dataset				Evaluation Dataset
Method	Baby Cry	Glass Break	Gunshot	Average	Baby Cry	Glass Break	Gunshot	Average
	ER|F1	ER|F1	ER|F1	ER|F1	ER|F1	ER|F1	ER|F1	ER|F1
CRNN+TA [61]	*******	*******	*******	*******	0.25|87.4	0.05|97.4	0.18|90.6	0.16|91.8
CRNN+Attention [61]	0.10|95.1	0.01|99.4	0.16|91.5	0.09|95.3	0.18|91.3	0.04|98.2	0.17|90.8	0.13|93.4
Mult-Scale RNN [31]	0.11|94.3	0.04|97.8	0.18|90.6	0.11|94.2	0.26|86.5	0.16|92.1	0.18|91.1	0.20|89.9
R-CRNN [43]	0.09|***	0.04|***	0.14|***	0.09|95.5	*******	*******	*******	0.23|87.9
CRNN [62]	*******	*******	*******	0.14|92.9	0.18|90.8	0.10|94.7	0.23|87.4	0.17|91.0
1D-CRNN [63]	0.05|97.6	0.01|99.6	0.16|91.6	0.07|96.3	0.15|92.2	0.05|97.6	0.19|89.6	0.13|93.1
[64]	*******	*******	*******	*******	0.29|86.20	0.09|95.47	0.28|85.60	0.22|89.09
MTFA [21]	0.06|96.7	0.02|99.0	0.14|92.7	0.07|96.1	0.10|95.1	0.02|98.8	0.14|92.5	0.09|95.5
MS-FCN [29]	0.10|95.1	0.02|99.0	0.13|93.1	0.08|95.7	0.11|94.4	0.06|96.8	0.08|96.2	0.08|95.8
MSFF-Net [30]	0.10|94.97	0.03|98.60	0.11|94.43	0.08|96.00	0.10|94.82	0.05|97.59	0.08|95.97	0.08|96.13
MS-FCN+Hybrid-CE	0.12|93.9	0.20|99.0	0.12|94.0	0.08|95.6	0.11|94.4	0.05|97.4	0.06|96.8	0.07|96.2
SS-FCN+HDCi+Hybrid-CE	0.11|94.31	0.01|99.60	0.13|93.3	0.08|95.7	0.12|93.8	0.06|96.8	0.06|97.0	0.08|95.9

.

Observing Table 6, compared to MS-FCN with CE loss, after hybrid CE loss is used, on the development dataset, the performance is not improved significantly, while on the evaluation dataset, improvements of 1% and 0.4% are obtained for the class-wise average error rate and F1 score. This further suggests hybrid CE loss is superior to CE loss. As for the ability to model long short-term dependencies of MS-FCN and HDC-Inception, the results of MS-FCN with hybrid CE loss and SS-FCN with hybrid CE loss and HDC-Inception are shown in the penultimate and the last rows of Table 6. For the development dataset, the performance of both methods is similar, while for the evaluation dataset, the network based on MS-FCN achieves improvements of 0.01 and 0.3% in the class-wise average error rate and F1 score. Clearly, the two methods of obtaining long short-term dependencies do not show significant differences.

Observing all methods in Table 6, on the development dataset, for gunshot, MSFF-Net [30] achieves the best performance, and for glass break, SS-FCN with hybrid CE loss and HDC-Inception delivers the best performance. Moreover, 1D-CRNN [63] and MTFA [21] achieve high performances for the baby cry event. On the evaluation dataset, MS-FCN based on hybrid CE loss shows the best class-wise average error rate and F1 score. For gunshot, SS-FCN based on hybrid CE loss and HDC-Inception has the best performance. For baby cry and glass break events, MTFA [21] has the best performance. Although 1D-CRNN is better performing on the development dataset, it has a much lower score on the evaluation dataset and uses the ensemble strategy with high complexity. MTFA, while performing better on the development dataset, sees its class-wise average error rate and F1 score worsen by 0.02, 0.01, 0.07%, and 0.04% than the proposed method, respectively. Moreover, MTFA uses RNN to model time dependencies, processing data in a serial manner rather than a parallel manner. This results in a decrease in operating speed by approximately 20 times compared to similar fully convolutional networks [29,30], causing MTFA to require longer training times. MSFF-Net [30] and the proposed method in this manuscript are both improvements based on SS-FCN and MS-FCN [29]. Observing Table 6, both exhibit similar performance on the development dataset, while the proposed method shows improvements of 0.01 and 0.07% compared to MSFF-Net [30] on the evaluation dataset for the class-wise average error rate and F1 score, respectively.

5.2.2. TUT-SED 2017 Dataset

The TUT-SED 2017 dataset, extracted from real life, is used to evaluate industrial application value; the results are shown in

Table 7. Performance comparison. Dataset: TUT-SED 2017 dataset. The symbol “***” indicates that specific results are not available. HDCi: HDC-Inception. The best results are marked in bold face.

	Development		Evaluation
Method	ER	F1 (%)	ER	F1 (%)
SS-RNN [65]	0.61 ± 0.003	56.7	0.825	39.6
LSRM [66]	0.66	54.5	0.853	39.1
CNN [67]	0.81	37	0.858	30.9
DCASE2017 Baseline [58]	0.69	56.7	0.9358	42.8
MS-RNN [20]	0.604 ± 0.001	***	***	***
CRNN [68]	0.6	59	0.7914	41.7
SS-FCN [29]	0.5724 ± 0.008	61.02 ± 0.63	0.8132 ± 0.0145	47.03 ± 1.73
MS-FCN [29]	0.5714 ± 0.0097	61.21 ± 0.85	0.7843 ± 0.019	48.60 ± 0.74
MSFF-Net [30]	0.5805 ± 0.0047	59.76 ± 0.79	0.7519 ± 0.0074	49.93 ± 1.5
LocEnc100 [28]	***	***	0.90	49.86
attn200 [27]	***	***	0.681	49.6
MS-AttDenseNet-RNN [69]	***	***	0.76	49.6
MS-FCN+Hybrid-CE	0.5745 ± 0.0044	60.27 ± 1.28	0.7717 ± 0.0402	48.86 ± 0.91
SS-FCN+HDCi+Hybrid-CE	0.5893 ± 0.0135	59.84 ± 0.99	0.7662 ± 0.0209	49.62 ± 0.96

. Observing the second and fourth columns in the last row of Table 7, after hybrid CE loss replaces CE loss, the performance on the development dataset decreases slightly, while on the evaluation dataset, improvements of 1.26% and 0.26% are obtained for the class-wise averaging ER and F1 scores. This pattern is similar to the results from the TUT Rare Sound Event 2017 dataset. Overfitting occurs on the development datasets; however, some improvements are observed for the development dataset. The differing results on the development and evaluation datasets demonstrate that the proposed hybrid CE loss can alleviate overfitting. For HDC-Inception, when comparing its ability to capture long short-term dependencies against MS-FCN, the results are displayed in the last and second-to-last rows of Table 7. As the DCASE Challenge focuses on the performance of the evaluation dataset, the proposed HDC-Inception is more competitive. On the evaluation dataset, the method based on HDC-Inception shows improvements of 0.55% and 1.24% in class-wise error rate and F1 score, respectively, compared to the method using MS-FCN. This result is different from that of the TUT Rare Sound Event 2017 dataset. The different results indicate that both approaches have their advantages and disadvantages. After HDC-Inception and hybrid CE loss are used, compared with SS-FCN, SS-FCN+HDCi+Hybrid-CE achieves improvements of 4.7% and 2.59%, on class-wise ER and the F1 score. This indicates that the proposed method shows a significant improvement in this dataset.

In order to present the results comprehensively and compare the misclassification of different categories, confusion matrices based on quantity and proportion are constructed as shown in Figure 7. When constructing the confusion matrix, the segment lengths are set to 1 s, as segment-based metrics. As shown in Figure 7, for the brakes squeaking event, all samples are misclassified, with 79% of them being misjudged as the car. This result is similar to that obtained by the dataset developers, where the ER for the brakes squeaking was 98% and the F1 score was 4.1% [58], nearly a complete misclassification. Analyzing the reasons, as shown in Table 1, the brakes squeaking event has 52 samples with a total duration of 96.99 s, and only about 70 s is selected during the training process, resulting in an extremely small total duration for this event. Moreover, car engine running and car idling were also labeled as the car event. It is evident that car engine and car idling events have a high similarity to brakes squeaking, which is also the cause of the misjudgment. For the car event, the accuracy is relatively high. For the children event, all samples are misclassified, with 73% being misjudged as people walking. This result is similar to the dataset developers’ findings, where the ER for children was completely wrong and the F1 score was 0 [58]. The Car event has the second shortest duration of 350.99 s. Beyond that, during dataset partitioning, the authors divided children yelling and children talking into the children category, which also led to the majority of children samples being misclassified as people speaking and people walking. For the large vehicle event, the proportion of correct judgments is 49%, while the proportions of misclassifications as car and background are 21% and 17%, respectively. It is understandable that large vehicle and car events have a high similarity. For people speaking and people walking, the proportions of correct judgments are 39% and 55%, respectively. The two categories have a relatively high mutual misclassification rate, as it is common for people to walk while talking or talk while walking. For the background event, the proportion of correct judgments is 63%, with 23% misclassified as the car event. Overall, the proportions of misclassifications as car and people walking events are relatively high, due to the longer total durations of these two events on the training dataset, as car has the largest duration of 2471.14 s and people walking has the second-largest duration of 1246.95 s. Therefore, the primary reasons for the model’s misjudgment are inferred to be class imbalance and inter-sample similarity.

Compared to classical CRNN [62], methods based on the hybrid CE loss and HDC-Inception gain large improvements on both development and evaluation datasets. SS-FCN with hybrid CE loss and HDC-Inception reduces ER by 0.0107 and improves the F1 score by 0.84% on the development dataset, as well as reduces ER by 0.0152 and improves the F1 score by 7.92% on the evaluation dataset. For MS-FCN, a noticeable improvement is captured by the proposed method. Similar to MSFF-Net [30], the proposed method is an improved method based on MS-FCN [29], and both exhibit comparable performance. Ref. [69] is a method based on dense convolution and attention mechanisms, and its performance is similar to ours. Refs. [27,28] are methods based on Transformer or self-attention mechanisms. Specifically, ref. [28] discussed the impacts of two self-attention mechanisms, time-restricted and full-length, on polyphonic sound event detection, suggesting that polyphonic sound event detection benefits more from features within a limited time frame; hence, the time-restricted self-attention prevails. On the TUT-SED 2017 datasets, Ref. [28] achieved an ER of 0.9 and an F1 score of 49.86%. Its F1 score is slightly superior to that of the proposed method, which is 49.42%, but the ER is excessively poor compared to our 0.7662. Similarly, Ref. [27] employed a self-attention-based method. During training, Ref. [27] incorporated the DCASE 2021 dataset, resulting in an ER of 0.6810 and an F1 score of 49.6%. Although their F1 score is comparable to ours, their ER is 8.52% better than ours. However, the question remains whether the improvement is from data augmentation or the Transformer/self-attention architecture. If it originates from architecture, why does [28] not show any improvement?

This issue has been discussed in the fields of image processing and natural language processing (NLP). Researchers have consistently indicated that Transformer/self-attention-based methods require large datasets to demonstrate their advantages, which are not as evident in smaller datasets [70,71,72]. In image processing, Ref. [70] suggested that Vision Transformers struggle to effectively learn inductive biases in small datasets. Similarly, in the NLP domain, Ref. [71] found that switch Transformers did not exhibit significant advantages in clinical narrative classification when applied to small datasets. Both [71,72] emphasized the necessity of special strategies such as data augmentation, pre-trained models, and knowledge distillation when using Transformer models on small datasets. This also explains why, in signal classification and the detection domain, Transformer-based models are predominantly used with large datasets or weak labels, such as AST-SED [23], CNN-Transformer [24], and SEDT [25]. For this reason, in polyphonic sound event detection, Transformer/self-attention-based methods employ data argumentation or knowledge distillation frequently, as seen in Ref. [27] with the DCASE 2021 dataset. Under unequal conditions and without the utilization of data augmentation or external datasets, the proposed method exhibits comparable performance to methods based on Transformer or self-attention mechanisms on the TUT-SED 2017 dataset.

5.2.3. TUT-SED 2016 Dataset

The TUT-SED 2016 dataset is small, containing audio recordings of 78 min with 17 types of events, and it is used to validate the performance of small real-world datasets. For MS-FCN, switching the loss function from CE loss to hybrid CE loss, for the performance of the development dataset, there is no significant change, and for the evaluation dataset—the ER and F1 scores achieve improvements of 0.0041 and 1.64%. This change in performance is similar to that of the TUT Rare Sound Event 2017 dataset and the TUT-SED 2017 dataset, validating the effectiveness of the proposed hybrid CE loss further. Comparing MS-FCN and HDC-Inception, as shown in the last two rows of

Table 8. Results on the TUT-SED 2016 dataset. The symbol “***” indicates that specific results are not available. HDCi: HDC-Inception. The best results are marked in bold face.

	Development		Evaluation
Method	ER	F1 (%)	ER	F1 (%)
GMM [59]	1.13	17.9	***	***
FNN [15]	1.32 ± 0.06	32.5 ± 1.2	***	***
CNN [19]	1.09 ± 0.06	26.4 ± 1.9	***	***
RNN [19]	1.10 ± 0.04	29.7 ± 1.4	***	***
MS-RNN [20]	0.82  ±  0.01	31.5  ±  0.8	***	***
CRNN [19]	0.95  ±  0.02	30.3  ±  1.7	***	***
SS-FCN [29]	0.7807 ± 0.0088	41.62 ± 0.94	0.9367 ± 0.0084	26.66 ± 1.46
MS-FCN [29]	0.7780 ± 0.0127	42.02 ± 1.85	0.9328 ± 0.0411	25.39 ± 3.05
MSFF-Net [30]	0.7806 ± 0.0103	40.85 ± 0.0179	0.9264 ± 0.0099	25.95 ± 0.0156
MS-FCN+hybrid-CE	0.7769 ± 0.0234	40.68 ± 0.02.23	0.9287 ± 0.0218	27.03 ± 1.92
SS-FCN+HDCi+hybrid-CE	0.8078 ± 0.0067	37.11 ± 1.41	0.9289 ± 0.0118	26.33 ± 1.40

, MS-FCN shows better performance on the development dataset, while each performs similarly on the evaluation dataset. Observing the last row and the fourth row from the bottom of Table 8, the transition from MS-FCN to the method based on HDC-Inception and hybrid CE loss results in improved ER and F1 scores by 0.0039 and 0.94% on the evaluation dataset, while there is a significant decline on the development dataset. Compared with MSFF-Net [30], also based on MS-FCN [29], the proposed method shows an improvement of 1.08% in the class-wise average F1 score on the evaluation dataset, which means that the proposed improvement measures of HDC-Inception and hybrid CE loss are more effective. As shown in Table 8, the proposed method significantly outperforms the existing CRNN [19] and MS-RNN [20] methods, demonstrating greater performance improvement and high adaptability to small datasets.

The summary of the experimental results is as follows: (1) Observing MS-FCN vs. MS-FCN+Hybrid-CE, (i) for the TUT Rare Sound Event 2017 evaluation dataset, MS-FCN+Hybrid-CE achieves improvements of 1% and 0.4% for the ER and F1 scores, as shown in Table 5. (ii) For the TUT-SED 2017 evaluation dataset, ER and F1 scores achieve improvements of 1.26% and 0.26%, as shown in Table 6. (iii) And for the TUT-SED 2016 evaluation dataset, improvements of 0.41% and 1.64% are obtained, as shown in Table 7. (2) Observing SS-FCN vs. SS-FCN+HDCi+Hybrid-CE, (i) for the TUT Rare Sound Event 2017 evaluation dataset, SS-FCN+HDCi+Hybrid-CE achieves improvements of 4% and 1.91% for the ER and F1 scores, as shown in Table 4. (ii) For the TUT-SED 2017 evaluation dataset, improvements of 4.7% and 2.59% are obtained on class-wise ER and F1 scores, as shown in Table 6. (iii) And for the TUT-SED 2016 evaluation dataset, ER improved by 0.78%, while the F1 score remained consistent. The results indicate that the proposed HDC-Inception and hybrid CE loss are effective, achieving a maximum improvement of 4.7%. Compared with HDC-Inception, the hybrid CE loss exhibits more clear improvements, resulting in a 3% enhancement in performance as shown in Table 4.

5.2.4. Computational Costs

Model size and FLOPs (floating point operations) are selected to measure the complexity of models. The model size reflects the total number of parameters in a neural network; a larger model size means higher memory usage. FLOPs refer to the number of basic operations during a single forward or backward propagation in a neural network, reflecting the computational load of the network. The larger the FLOPs, the more computational resources and time the neural network requires, and the higher the complexity of the model. As shown in

Table 9. Computational costs of the proposed networks. Model size: the number of parameters of the model. FLOPs: the number of basic operations during a single forward or backward propagation in the networks.

Method	SS-FCN [29]	SS-FCN+HDCi+Hybrid-CE	MS-FCN [29]	MS-FCN+Hybrid-CE
model size	2.5445 MB	2.6761 MB	3.8741 MB	3.8741 MB
FLOPs	0.9936 G	1.0086 G	1.171 G	1.171 G

, SS-FCN [29] exhibits the most optimal model size and FLOPs. In contrast, MS-FCN [29] possesses the largest model size and FLOPs. The proposed hybrid CE loss is used as a loss function solely in models; hence, the model size and FLOPs of MS-FCN+hybrid CE loss are identical to those of MS-FCN. The proposed HDC-Inception, by introducing additional branches to capture long short-term dependencies, does not significantly increase the model size and FLOPs. The model size and FLOPs of HDC-Inception are less than those of MS-FCN. This indicates that—-compared to MS-FCN—the proposed HDC-Inception is more favorable in terms of model size and computational complexity.

6. Conclusions

To extract richer long short-term dependencies, HDC-Inception is proposed, which combines the advantages of HDC [34] and the Inception module [32,33], extracting multi-scale temporal dependencies and alleviating the “gridding issue” in dilated convolution. To select support vectors and focus on them to guide training, inspired by the support vector machine (SVM), the soft margin CE loss is proposed. This loss function can discriminate between support vectors and non-support vectors, guide the training by focusing on support vectors, and manage multiple classification tasks, which have wide applications. To address the problem of insufficient support vectors in the soft margin CE loss, the hybrid CE loss is proposed. Hybrid CE loss combines the advantages of CE loss and soft margin CE loss, guides the training with all samples, and gives weight to support vectors. After the hybrid CE loss is selected, the ER and F1 scores are improved by 3% and 1.51%, respectively, on the TUT Rare Sound Event 2017 dataset. When hybrid CE loss and HDC-Inception are selected, the ER and F1 scores are improved by 4% and 1.91%, respectively. On the TUT-SED 2017 dataset, after both are selected, the ER and F1 scores are improved by 4.7% and 2.59%, respectively. On the TUT-SED 2016 dataset, the ER and F1 scores also improved. This indicates that the proposed HDC-Inception and hybrid CE loss can effectively enhance the performance.

Through theoretical analysis and experimental verification, we have reason to believe that the proposed HDC-Inception and hybrid CE loss have great potential and are worth further exploration. In particular, the hybrid CE loss and soft margin CE loss can adaptively screen samples, thereby selecting more valuable samples for training. Moreover, the proposed HDC-Inception shows strong compatibility with other network structures, and soft margin CE loss and hybrid CE loss are suitable for all classification problems. We will further research the sample selection mechanisms of the soft margin CE loss, striving to link sample selection with the performance of the model. We expect it will play a role in industrial application scenarios.