Identifying the Acoustic Source via MFF-ResNet with Low Sample Complexity

Abstract

Acoustic signal classification plays a central role in acoustic source identification. In practical applications, however, varieties of training data are typically inadequate, which leads to a low sample complexity. Applying classical deep learning methods to identify acoustic signals involves a large number of parameters in the classification model, which calls for great sample complexity. Therefore, low sample complexity modeling is one of the most important issues related to the performance of the acoustic signal classification. In this study, the authors propose a novel data fusion model named MFF-ResNet, in which manual design features and deep representation of log-Mel spectrogram features are fused with bi-level attention. The proposed approach involves an amount of prior human knowledge as implicit regularization, thus leading to an interpretable and low sample complexity model of the acoustic signal classification. The experimental results suggested that MFF-ResNet is capable of accurate acoustic signal classification with fewer training samples.

1. Introduction

Acoustic signal classification plays a central role in acoustic source identification [1], including many applications, such as marine animal [2], civil aviation aircraft [3], and noise source identification [4,5]. Generally, this problem can be solved in two different ways, the first one is to classify acoustic signals with manual design features, the model of which is low expressive but easy to interpret. The other is learning with deep representation features, the model of which is high expressiveness but calls for a huge amount of sample complexity. This manuscript provides a novel features fusion model with bi-level attention mechanism, which has a feature enhancement module and a feature fusion module. This approach involves an amount of prior human knowledge as implicit regularization, thus leading to an interpretable and low sample complexity model of the acoustic signal classification.

Classic sound signal identification methods include GMMs (Gaussian mixture models), SVMs (support vector machines), HMMs (hidden Markov models), and other classic machine learning methods [6]. They include manual identification and are therefore able to generate accurate results. Stephen So et al. have worked on sound classifiers based on Gaussian mixture models (GMMs) and leverage MFCC to improve the identification performance of the algorithms [7]. Bittle and Duncan et al. incorporate the support vector machine (SVM) sound target classification technology in sound source identification and classify the signal based on the similarity between the unknown signal and the model training data [8]. Further, Daniele Salvati et al. propose a support vector machine (SVM) with a kernel function of RBF and construct a weighted MVDR beamformer, which works well in near-field single sound source identification. However, the array gain of the conventional beamformer is limited. In order to maximize the array gain, Capon proposed the MVDR beamformer in 1969 [9,10]. Walters et al. propose applying the random forest method to acoustic classification so as to achieve training on large-scale datasets [11]. Zilli Parson et al. apply the hidden Markov model (HMM) to acoustic analysis and classification because this model is able to fuse the sound signal and the temporal details in the sequence [12]. The methods described above, however, because of their incapability of automatic feature extraction, typically perform poorly when recognizing new signals; they are still needful of a new algorithm. This restricts their scope of application and diminishes their generalization ability.

In recent decades, deep learning [13] has provided state of the art performance in lots of areas including text [14], images [15], and sound data analysis [16], because it can solve problems in a big range of complex pattern recognition. It has equally given excellent results in the classification of acoustic signals. Pirotta et al. explore applying unsupervised learning to acoustic signal classification and make grouping based on the similarity of features between data (cluster algorithm) [17]. Goeau and Mac Aodha have studied the convolutional neural network technology without feature pre-processing and propose identifying features directly from the spectrogram data, which avoid the noise-sensitive feature extraction stage [18]. Boqing Zhu et al. propose a CNN architecture, where the network is integrated for multi-time resolution analysis and multi-level feature extraction, which solves the multi-scale problem in environmental acoustic classification [19]. Oikarinen et al. introduce a convolutional neural feedforward network, where the original spectrogram is taken as input, enabling classifying the source and type of acoustic signals in a noisy environment [20]. However, in real applications, it is expensive or even impossible to gain varieties of samples, such as medical imaging and 3D reconstruction. In addition, these traditional deep neural networks do not involve manual domain knowledge, which is prone to overfitting and poor interpretability even under heavy data augmentations.

Therefore, researchers are interested in improving deep learning models and with fewer training samples. A straightforward idea is data enhancement techniques [21]. Several training data generators have been proposed, such as deep learning under conditions of Gaussian noise [22,23] or object attributes [24]. Nonetheless, when being trained on small sample data, such data generators often fare poorly [25]. The second solution is to optimize the neural network algorithm. Erhan et al. investigate unsupervised pre-training in deep learning and substantiate the generalization effect of pre-trained network weights experimentally [26]. Yu-Xiong Wang et al. suggest a method that works, by means of unsupervised data sources to improve the overall transferability of supervised CNNs, thereby permitting the identification of new classes from a small number of samples [27]. Mishkin and Matas propose the initialization of variances of layer sequence units, whereby the weight of each convolutional layer is initialized through an orthogonal matrix so that the weights are normalized into unit variance [28]. The third method is to adjust the structure of the neural network model. Qianru Sun et al. propose a meta-transfer learning (MTL) method, which is able to transfer the weights of a deep neural network with fewer training samples [29]. Mengye Ren et al. proposes a semi-supervised small sample size learning mode to learn how to make use of unlabeled samples, which improve model performance under semi-supervision [30]. Mark Woodward proposes an active learning method that combines reinforced learning with small sample-size learning to achieve higher prediction accuracy than pure supervised models [31].

The overall goal of this manuscript is to identify the acoustic source with low sample complexity, which intends to overcome the bottleneck of the current deep learning methodology. In this study, a novel bi-level attention mechanism was proposed to fuse the handcraft features and deep representation features in order to include more prior human knowledge, thus a low sample complexity to train the neural network. The main contributions of this paper are the construction of two sub-networks, named feature enhance module and feature fusion module, respectively. In the feature enhance module, this study applies scaled dot-product attention to local level enhancing of both handcraft features and deep representation features. In the feature fusion module, the manuscript modifies the self-attention to obtain a global fusion of the enhanced features.

To verify the model we proposed, this study was tested on a public dataset named Esc: Dataset for environmental sound classification [32], and another dataset was collected by this research. This study chose four different models for comparison, such as the classical rResNet, the state of art method EnvNet [33], EnvNet v [34], GoogLeNet [35], MFF-ResNet (without bi-level attention), and MFF-ResNet (without rResNet) for ablation experiment. The experiment suggested that the MFF-ResNet performs quite well on a small sample training set of sound source classification compared with baseline methods.

This paper is so structured that in Section 2, the current deep learning methods for processing spectrograms are discussed. Section 3 presents the MFF-ResNet model and describes its structure. Section 4 describes the experiment and gives an analysis of the experimental results. Section 5 provides a summary of this paper.

2. Background

2.1. Data Pre-Processing

As shown in Figure 1, the sound data are collected by the acoustic signal acquisition equipment with different channels of $\mathrm{h}$. The original signals are waveforms represented in the time domain. To analyze the power spectrum properties, it is necessary to convert the original signals with certain integral transform, such as Fourier transform or wavelet transform. Due to the power, spectrum contains the characteristic information of the source signal in different frequency bands at different times, similar as image processing, and a deep convolutional network can be applied in high-level feature representation.

In the field of sound signal processing, MFCC (Mel frequency cepstrum coefficient) is often extracted during audio analysis, but this causes a big loss of sound details. After the birth of deep learning, thanks to the powerful feature extraction capability of deep neural networks, it becomes possible to allow the neural network to extract more robust features by feeding the log-Mel spectrum (or power spectrum) information directly into the neural network for training, as shown in Figure 2.

2.2. Related Works

The diagrammatic sketch in Figure 3 depicts the general procedure for acoustic signal classification. Firstly, a microphone array consisting of high-sensitivity omnidirectional sound sensors collected sound signals from different sound sources. After data pre-processing, manual design features and log-Mel spectrogram were extracted to obtain the manual design classification features in the form of number of manual features, number of frames, or number of channels and the log-Mel spectrogram features in the form of (frequency, time_steps, channel). Secondly, a deep learning training network suitable for acoustic signal data was constructed, such as ResNet. Finally, the network model was trained and then tested for its performance through experiments.

It is apparent that the main issue in acoustic signal classification is designing a neural network to learn, represent, and fuse data features. Due to the strong expressive ability of neural network, deep learning models have achieved extraordinary performance compared with traditional signals’ processing methods. However, experiments showed that after the network has reached a certain depth, a further greater number of layers do not lead to further improvement in classification performance but rather leads to slower network convergence and worse classification accuracy. One of the most widely used approaches is named ResNet [36], which was proposed to resolve this problem based on the concept of residual representation. The schematic diagram of the residual block and a typical ResNet is shown in Figure 4a.

The signals put in ResNet can be directly connected to an earlier layer through a shortcut, so that a large number of effective features are still retained at the end of the network, and the feature signals in the data are fully exploited, which can maintain a strong increase momentum in accuracy with an increasing depth as shown in Figure 4b. This makes the ResNet classification network the most widely used CNN feature extraction network.

Recently, Anam Bansal uses a deep neural network method to verify its superiority in ambient sound classification performance [37]. Ref [38] proposed an attention-based ResNet, which used attention-guided deep model to efficiently learn spatial-temporal relationships that exist in the spectrogram of a signal. In this study, we aim to train ResNet with low sample complexity, thus using bi-level attention mechanisms to enhance and fuse deep representation features with the handcraft features, in order to include more prior knowledge.

3. Methodology

3.1. The Architecture of MFF-ResNet

To classify acoustic signals with low sample complexity, this study proposes a novel bi-level attention mechanism to fuse the handcraft features and deep representation features in order to include more prior human knowledge, thus a low sample complexity to train the neural network. As depicted in Figure 5, the log-Mel spectrogram features and manual design features fusion ResNet (MFF-ResNet) are proposed.

As the input of the proposed network, log-Mel spectrogram features and manual design features are included in this fusion model. Then, the deep represented Mel spectrogram is pertained with fine-tuned rResNet. Sequentially, deep represented features and manual design features are fused with a bi-level attention module. Finally, a feature classify module is applied for the downstream task.

3.2. Manual Design Features

Manual design features are based on the feature engineering method [39]. Manual design is completed beforehand and features are extracted from audio signals. Manual design features are most concerned with the overall recognition of acoustic signals, such as pitch, tone, and silence rate. By means of manual design features, meaningful data features are derived from the data. High-quality features help to improve the overall performance and accuracy of this model. Features basically concern basic problems, and it is necessary to design features related to scenes, problems, and domains.

The open-source audio analysis software Essentia [40], more specifically, it’s feature extractor Freesound, was used to extract manual features from the acoustic data. First, the audio file was framed such that a 10-s sound signal was divided into 500 frames, each frame signal being 40 ms, with 20 ms of overlap between adjacent frames.

Feature extraction was performed with Freesound: For each frame signal in a single channel, the Freedsound’s feature [41] extraction function (parameters were set to the default values) was used to extract 400 features, as shown in the

Table 1. The features picked up by the Freesound feature extractor.

Feature Name	Dimension	Feature Name	Dimension
Bark bands energy	52	Tonal signature	35
Erb bands energy	45	Pitch signature	35
Mel bands energy	64	Silence rate	35
Spectrum signature	28	MFCC	42
HPCP	38	GFCC	26

. From each sound sample, Freesound manual design acoustic source features were extracted in the form of (number of manual features, number of frames, number of channels).

3.3. Deep Representation Features with Fine-Tuned rResNet

3.3.1. rResNet

The log-Mel spectrogram data in the form of (frequency, time_steps, channel) were split along the frequency axis into the first half low-frequency band and the second half high-frequency band. Because of that, the high-frequency and low-frequency parts of the data had different forms of characteristics. Inspired by wavelet nets, the ResNet in the frequency framework of this study is improved. The frequency energy is separated into a high-frequency part and a low-frequency part. The convolution can be the type of filter that is desired; this study used skip connection only for the high-frequency part, and then this modified ResNet was named as rResNet, which is shown in Figure 6.

The convolution kernel size of these convolutions in the network was 3 × 3; the padding was set to “same” so that each time the data passed through the convolution layer only its channel number was changed, but the frequency axis and the dimensions of the time axis were preserved. For the sake of subsequent frequency axis dimension fusion, no down-sampling was performed. After the pre-trained network, the outputs from the two paths were fused by concatenation operations on the frequency axis, to revert to the frequency dimension of the original data, which completed the frequency fusion.

3.3.2. Pre-Training and Fine-Tuning Technology

A common and very efficient way to apply deep learning to small sample datasets is to use a pre-trained network [42,43,44,45,46]. A pre-trained network is a model that has already been trained on a large dataset. If the original dataset is large and general enough, the feature space hierarchy learned by the pre-trained network can work effectively as a general model for acoustic analysis. Therefore, these features can help solve various acoustic signal classification problems, even if these new problems involve classes that are completely different from the original task.

In this section, this manuscript represents a pre-training and fine-tuning technology for the backbone network named rResNet. Pre-training and fine-tuning technology was widely used in the acoustic scene and event detection classification task, such as DCASE. As shown in Figure 9, the approach of this study first keeps the convolution part only of the network for feature extraction, so that the classifier part of the pre-trained network was discarded but uses a new classifier adapted to this training data. Inspired by the fact that the generality (and reusability) of feature extraction of a convolutional layer depends on the depth of the layer in the model. The deeper the layer in the model, the more abstract the extracted concept is. Since the new dataset differed much from the dataset used to train the original model, only the beginning part of the model was used for feature extraction with the deeper layers being trained along with the newly included classifier. Thus, this study next freezes the low-level convolutional to fine-tune high-level convolutional. The convolution part of the pre-trained network ResNet contained 12 convolutional layers, the first 8 of which were of frozen weights. The weights of the rest 4 convolutional layers were unfrozen. By training the weights of these deeper convolutional layers, the abstract representation of the model was made more relevant to the classification problem in question.

3.4. Feature Fusion Based on Bi-Level Attention

In order to encode prior human knowledge as implicit regularization, this study develops a robust and an efficient features fusion strategy based on bi-level attention. The motivation lies on the manual design features providing an abundant interpretation and deep representation of features providing an expressive abstraction of the given acoustic signal. Thus, fused features contain much useful prior information and a good description of the acoustic signal features.

The features fusion strategy of this study consists of two modules, named features enhancement module and feature fusion module, respectively. Both of these modules are constructed based on the attention mechanism. Inspired by the selective visual mechanism of humans, the attention mechanism can filter out the most influential parts from a large amount of information according to certain queries.

3.4.1. Features Enhancement Module

Without loss of generality, suppose manual designed features are denoted as ${\mathrm{x}}_{\mathrm{h}}\in R^{p\times T\times C_1}$, in the form of (features, time-steps, channel) and the deep representation features are denoted as ${\mathrm{x}}_{\mathrm{d}}\in R^{q\times T\times C_2}$. As shown in Figure 7, in order to enhance deep representation features ${\mathrm{x}}_{\mathrm{d}}$, the Features Enhancement Module firstly applies $1\times 1$convolution to adjust the ${\mathrm{x}}_{\mathrm{h}}{\prime }^{\mathrm{s}}$ input channel numbers and then uses an interpolation method (resampling) to adjust the dimension, thus leading to modified manual design features as ${\mathrm{x}}_{{\mathrm{h}}^{\prime }}\in R^{q\times T\times C}$. This paper use ${\mathrm{x}}_{{\mathrm{h}}^{\prime }}$as quires to reweight deep representation features ${\mathrm{x}}_{\mathrm{d}}$into ${\mathrm{x}}_{\mathrm{d}}^{\prime }$, in order to give greater weight to important components and give less weight to unimportant components. The weights of the components are obtained by the softmax function of the inner product between ${\mathrm{x}}_{\mathrm{d}}$ and${\mathrm{x}}_{{\mathrm{h}}^{\prime }}$. The mathematical expression is:

$$e_{d_i}={\mathrm{x}}_{{\mathrm{h}}^{\prime }}^T{x}_d{}_i$$

(1)

$$w_{d_i}=\frac{e_{d_i}}{{\sum }_{i=1}^q{e}_{d_i}}$$

(2)

$${x^{\prime }}_d={\mathrm{w}}_{\mathrm{d}}^T{x}_d$$

(3)

It was noted here that there are no additional parameters involved in this module. Similarly, this study can enhance manual design features ${\mathrm{x}}_{\mathrm{h}}$ into ${{\mathrm{x}}^{\prime }}_{\mathrm{h}}\in R^{p\times T\times C}$.

3.4.2. Features Fusion Module

In the Feature Fusion Module as shown in Figure 8, this study slightly modifies the multi-head attention in order to fuse the enhanced manual design ${{\mathrm{x}}_{\mathrm{h}}}^{\prime }$ and deep representation features ${{\mathrm{x}}_{\mathrm{d}}}^{\prime }$. Instead of computing attention only once, the multi-head mechanism runs multiple scaled dot-product attention (SDPA) in parallel. However, the original SDPA requires the same length of input for computation. Thus, this study first concatenates ${{\mathrm{x}}_{\mathrm{h}}}^{\prime }$ and ${{\mathrm{x}}_{\mathrm{d}}}^{\prime }$ into a feature vector $\mathrm{x}=\left[\begin{array}{c}{{\mathrm{x}}_{\mathrm{h}}}^{\prime }\\ {{\mathrm{x}}_{\mathrm{d}}}^{\prime }\end{array}\right]\in R^{\left(p+q\right)\times T\times C}.$ For each channel $\mathrm{j}$, this paper uses a Matlab-like notation as $\mathrm{x}\left(:,:,\mathrm{j}\right),\left(\mathrm{j}=1,2\cdots \mathrm{C}\right)$. Sequentially, “Softmax” functions are applied to compute each ${\mathrm{head}}_{\mathrm{j}}\in R^{\left(p+q\right)\times s_2}$ as follows

$$hea{d}_j=\mathrm{softmax}\left(\frac{\left(x\left(:,:,j\right)W_{kj}\right){\left(x\left(:,:,j\right)W_{qj}\right)}^T}{\sqrt{s_1}}\right)(\mathrm{x}\left(:,:,j\right)W_{vj}$$

(4)

with $W_{kj}\in R^{T\times s_1}$, $W_{qj}\in R^{T\times s_1}$, and $W_{vj}\in R^{T\times s_2}$ as all learnable parameters.

Finally, the independent attention outputs are simply concatenated and linearly transformed to the expected dimension, thus the multi-head attention can be described as:

$$\mathrm{Multi}-\mathrm{Head}=\left[{\mathrm{head}}_1,\cdots ,{\mathrm{head}}_{\mathrm{j}},\cdots {\mathrm{head}}_{\mathrm{C}}\right]{\mathrm{W}}_0\in R^{p\times s}$$

(5)

Here, ${\mathrm{W}}_0\in R^{\left(s_1+s_2\cdots +s_c\right)\times s}$are also learnable parameters.

3.5. Feature Classifier Module

After being processed by the Feature Fusion Module, the fusion feature was input into the feature classifier. The feature classifier was comprised of two convolutional layers with a kernel size of 1 × 1 and subsequent additional layers. The number of convolution kernels of the first layer was 48, and the number corresponding to the second layer was reduced to the number of classes. What followed were the batch normalization layer, the global average pooling layer, and softmax. The two 1 × 1 convolution layers acted as a two-layer fully connected neural network. Finally, the softmax activation layer assigned weights to the contribution of each channel to arrive at the probability of the sample in each class.

4. Experiment and Result Analysis

4.1. Acoustic Dataset Collection (LSE)

In the experiment to verify the proposed model, this paper built a spherical acoustic sensor array of MicW i436 omnidirectional recording microphones. An acoustic signal hardware acquisition system was created using the hardware acquisition circuit module at the back end to collect multiple sound source signals, as shown in Figure 8. This study collected acoustic signal data from eight different sound source scenes, each source containing 1000 samples to a total of 8000 sample datasets. Each sample lasted 10 s, as shown in Figure 10a–d.

The detailed acquisition process is as follows,

(a)

Data acquisition: Firstly, this study connected the source of the excitation device with an external instrument (such as a laptop). Secondly, this study controlled the two kinds of sound source excitation devices to generate sound signal and noise, respectively, in order to guarantee the signal-to-noise ratio of less than −10 dB. It can be noted here, the source signals of sound include eight categories, such as airplane, bus, footstep, car, piano, gun, dog, and vocals. Each kind of sound source signal is a prepared audio file. The Gaussian white noises are immerged into sound signals according to certain signal-to-noise ratio setting. Finally, this study controls the microphone array to collect multi-channel acoustic signals and transmit them to the storage device.

(b)

Data slicing: After the data acquisition, the acoustic signal is sliced and processed into multiple eight-channel audio short signals with duration of 10 s.

(c)

Data preparation: After slicing the collected data, this study stores each kind of sound source into an intendent file, respectively. For each kind of sound sources, 2000 samples were collected. Therefore, there are totally 16,000 sound signal samples in the LSE (low-sample environmental) dataset. The proportion of the training set, validation set, and test were allocated as 6:2:2. The acoustic classification network model is learned and trained on the training dataset of this dataset.

Figure 10. Acquisition of acoustic signals: (a–c) are directional speakers, rigid ball microphone array, and acoustic signal acquisition equipment. (d) is the collected 10 s acoustic signal expressed in python.

Figure 10. Acquisition of acoustic signals: (a–c) are directional speakers, rigid ball microphone array, and acoustic signal acquisition equipment. (d) is the collected 10 s acoustic signal expressed in python.

4.2. Experiment

4.2.1. Ablation Experiment Based on LSE Dataset

Manual design feature extraction and log-Mel spectrogram extraction were performed on the dataset. First, this study performs log-Mel spectrogram extraction on the audio signal, as shown in Figure 11, and manual design feature extraction. In this experiment, a window size of 2048, hop length of 1024 was chosen and 128 n-mels, using the librosa implementation. The resulting spectrograms consisted of 469 time samples and 128 frequency bins. This study calculated the log-Mel deltas and delta-deltas without padding, which reduced the number of time samples to 461. Additionally, the labels (eight folds) were converted into one hot vector using one-hot-encode method.

These features were then loaded into MFF-ResNet for training. Stochastic gradient descent was included in the training. The amount of batch processing was 32, the loss function was binary-cross entropy, the optimizer was SGD, and the number of epochs was set to 500. The learning rate reset scheduling method was used so that, after completing 3, 8, 18, 38, 128, and 256 iterations, the learning rate was reset to the maximum value of 0.1. After that, the rate attenuated to 0.00001 in accordance with the cosine function. To compare the network performance, the pre-trained network rResNet was used to complete a training of 500 iterations on log-Mel spectrogram features.

To verify the generalization ability of the proposed model, in this study, K-folds cross-validation technique is used to analyze the average experimental results on the test dataset. Specifically, the dataset is divided into K parts. Each subset is disjointed, and the size is the same. One part is selected from the K parts as the verification set, and the remaining K-1 parts are used as the training set. In this way, K separate model training and verification are carried out. Finally, the K verification results are averaged as the verification error of this model. It can be noted here, the K of this study is chosen at 10.

Table 2. Performance comparison under low sample complexity based on the LSE dataset.

Model	Loss	Precision (%)	Recall (%)	F1-Score (%)	Final Accuracy (%)	Epochs of Model Fitting
MFF-ResNet	0.34	89.21	91.10	90.75	90.15	160
rResNet	0.60	77.56	76.39	77.98	76.28	210

shows the ablation result based on MFF-ResNet model and the rResNet (without bi-level attention) based on the collected LSE dataset. Six measures including loss, precision, recall, final accuracy, and epochs of model fitting are used to evaluate the model performance. The result shows bi-level attention can improve the rResNet a lot in general.

Figure 12 shows the classification accuracy and loss corresponding to MFF-ResNet and rResNet. It can be seen that with MFF-ResNet, the classification accuracy stabilizes at 90.15% in the end, and the loss value is as low as 0.34; the classification accuracy of rResNet reaches 76.28%, and the loss value is as low as 0.60. With MFF-ResNet, convergence happens after 160 iterations, which is faster than rResNet’s 210 iterations; and MFF-ResNet is capable of classification accuracy, as shown in Table 2. It can be seen that MFF-ResNet can complete an effective and accurate classification and recognition of the sound characteristics of different sound source targets.

To examine the performance under different training data sizes, this study created a data subset of 1000, 1500, …, 4500, 5000, …, 7500, and 8000 training samples out of the 8000 sample datasets. The accuracy of different models trained on different sample sizes, as revealed by training samples of different data subsets in the comparative experiment, is shown in Figure 13.

Figure 13 depicts the change in classification accuracy finally achieved by each model trained on different training sample sizes. The results show that, compared with rResNet, MFF-ResNet performs better in each data subset and shows better recognition performance in small sample data subsets, such as 1000, 2000, 3000, and 4000 samples. Additionally, when manual design features, pre-trained network rResNet, and bi-level attention were used, the model achieved high recognition accuracy with a small number of training samples. The incorporation of manual design features, the contribution of pre-trained network rResNet, and the bi-level attention have so increased the features available to the model that even on a small number of training samples for MFF-ResNet was still able to achieve a good recognition accuracy.

4.2.2. Comparison with the State of Art Methods Based on ESC10

In this section, this study is going to evaluate the performance of MFF-ResNet based on an open data set named ESC-10. ESC-10 is a labeled set of 400 environmental recordings (10 classes, 40 clips per class, 5 s per clip). It is a subset of the larger ESC-50 dataset.

Compared methods include EnvNet, EnvNet2, and GoogLeNet. The results for each method are evaluated using four indicators, such as precision, recall, F1-score, and accuracy. Average test results also used the K-fold cross-validation technique.

Table 3. Comparison of algorithms based on ESC10 datasets (the best performance is underlined).

Model	Precision (%)	Recall (%)	F1-Score (%)	Accuracy (%)
EnvNet [33]	89.50	87.80	89.93	88.10
EnvNet v2 [34]	90.36	89.98	91.10	91.30
GoogLeNet [35]	85.65	84.70	85.20	86.00
MFF-ResNet	95.17	88.66	95.39	94.25

demonstrated that the method proposed in this paper exhibits obvious advantages on average.

The confusion matrix of experimental results on the ESC10 dataset is shown in Figure 14. It can be seen that the network is more sensitive to instantaneous sharp sound signals, so it is more accurate in recognizing dog bark and baby cry, and it is more accurate for sound signals widely distributed on the time domain axis. Less sensitive sound energies, such as rain and rooster, are mostly concentrated in the low-frequency end, making it difficult for the network to accurately identify.

5. Discussion and Conclusions

In sound source recognition and detection applications, data-driven convolutional neural network training with low sample complexity is a key requirement for deep learning in sound signal classification. Because in the field of sound source recognition, data collection is a costly process, which leads to a lack of large-scale sample sets. To alleviate such embarrassment, this paper proposes to fuse spectrogram features and manual design features with a novel attention-based model named MFF-ResNet. It seems bi-level attention can achieve both local enhancement and global fusion of the input signals.

The proposed classification model is flexible, able to handle training tasks based on both large and small databases. As part of future work, this algorithm can also be extended to other application fields, such as medical image recognition, face recognition and face camouflage, face matching and video, and face sketching and photo matching.