1. Introduction
Because they disperse seeds to keep forests healthy, western black-crested gibbons are essential to their habitats. Gaining a grasp of these animals’ calls and social structures can help safeguard the ecological balance by improving our comprehension of their effects on the ecosystem. Because gibbons are highly sensitive to environmental changes, alterations in the forest and climate change are reflected in their calls. Scientists can identify potential threats to the ecosystem early on and implement necessary conservation measures by keeping an ear out for gibbon cries. Finally, by identifying the calls of the western black-crested gibbon, researchers have been able to gain a better understanding of both the species and the environments in which they live. This knowledge has also helped to develop tools and vital information for the development of more successful conservation strategies. These studies support the preservation of the ecosystem’s overall health and equilibrium, in addition to aiding in the conservation of the apes themselves.
The continued monitoring and conservation of the western black-crested gibbon group will be an important element of ecologically sustainable development. The western black-crested gibbon has the largest group size, about 270 groups, distributed within China [
1]. Gibbon calls are distinctive and travel long distances [
2]; people often use gibbon calls to figure out where and how many wild gibbon colonies there are, so that the right steps can be taken to protect them [
3]. Western black-crested gibbons are already being watched every day through sentinel statistical surveys, where staff listen to, record, and look at their calls in some higher-elevation areas [
4,
5]. Naturally, this manual monitoring strategy requires a lot of labor and takes a lot of time, so it is unable to consistently meet the demands of online and timely monitoring [
6]. Since passive acoustic monitoring (PAM) came along, the western black-crested gibbon has also been used to test these systems [
7]. Add to this the fact that social animals like the western black-crested gibbon use a variety of sounds in their communities to convey messages and maintain social bonds. Thus, recognizing their calls can help scientists understand their social structure, communication methods, and social dynamics. This is crucial for the in-depth study and conservation of these species. By analyzing the calls of these gibbons, researchers can identify differences between individuals and groups. This helps researchers track the numbers and distribution of specific groups for more precise ecological monitoring and conservation efforts. However, there are several challenges in recognizing the calls of western black-crested gibbons, as follows:
Although passive acoustics can meet the requirements of western black-crested gibbon group monitoring, it would be labor-intensive to manually identify gibbon sounds from a large number of acoustic recordings;
Deep learning models can be used to effectively recognize the calls of the western black-crested gibbon. However, the western black-crested gibbon is located in remote areas, so it is difficult to obtain a large dataset of calls;
The task of training a deep learning network to distinguish between various call durations is challenging;
How to effectively extract features from input sound data and perform feature fusion.
Deep learning has recently begun to take center stage in passive acoustic monitoring as a result of its ongoing development. It has been shown that this manual processing of late acoustic data can be alleviated by using convolutional neural networks (CNNs) (LeCun et al. [
8,
9]) and that specific species in acoustic recordings can be identified by trained classifiers [
10,
11,
12,
13,
14,
15,
16,
17]. However, for these particular species, a good classifier needs a lot of reliable data. This is due to the fact that an inadequate dataset for a deep learning model will result in a reduction in the model’s ability to categorize data [
18]. Additionally, gathering these data takes a lot of time, especially for remote and rare species.
The best solution that comes to mind in this paper to deal with the challenge of the lack of a dataset is data augmentation [
19]. In order to create new, additional training data, data augmentation is a technique that involves applying one or more deformations to a set of annotated training samples or basalizing, time-shifting, and adding background natural noise to the input speech data [
16,
20,
21]. Due to the lack of a dataset, the augmented data can successfully prevent the data overfitting issue during the classification training process. The effectiveness of using data augmentation to enhance the model’s performance has been demonstrated in a number of examples. Salamon et al. [
22] uses four data augmentation methods, Time Stretching (TS), Pitch Shift (PS), Dynamic Range Compression (DRC), and Background Noise (BG), to improve the accuracy of ambient sound classification; Davis et al. [
23] used four methods proposed by Salamon et al. [
22] and five Linear Predictive Resonance Frequency Coefficients (LPCC) data expansion methods to effectively improve the accuracy of ambient sound classification. Therefore, reasonable data expansion can effectively improve the accuracy and generalization performance of the model. However, most sounds are altered in acoustic properties during the data augmentation process by standard techniques like pitch scaling and temporal stretching; hence, current techniques fail to sufficiently account for the unique characteristics of each sound. As a result, only when knowledge of the target sound is present can the augmentation procedure be chosen in a reasonable manner. The unique vocalizations of many animals may be lost or changed if the augmentation process is carried out blindly.
Currently, the most used solution in dealing with the problem of missing data is the Generative Adversarial Network [
24]. The network is frequently used for improving speech and image data, because it can produce realistic virtual data without changing the characteristics of the original sound data [
25,
26]. Even though GAN networks produce better results for data enhancement, they still have some flaws: the first is that the effectiveness of the generated speech data will be extremely low in the absence of training data; the second is that it ignores the frequency and time features present in the sound, because traditional GAN data enhancement methods are used to generate speech data from waveforms and spectrograms; the third is that most of the sound data comes from the outdoors, so the background noise contained therein can cause traditional GAN networks to fail to learn effective animal sound features. To address the limitations of traditional GAN networks, in this paper, we choose the state-of-the-art WaveGAN network [
27], which can effectively compensate for these shortcomings.
Still, there is room for improvement in the field of bioacoustic classification with respect to the current approaches. Convolutional neural networks usually require uniformly long sound inputs, yet sound signals might differ greatly in length. Sound clips must be trimmed or processed separately before being transferred to the network, because the time-frequency properties in each sound clip vary in length. It is likely that this approach will introduce some noise or lose some crucial audio data. Recurrent neural networks, on the other hand, are made to process sequential input. They do this by using iterative units, which continuously change their internal state in order to retrieve pertinent information from lengthier sound sequences. Convolutional neural nets (CNNs) are highly effective in classification tasks, giving spatial structure computation priority. They are therefore especially suitable for processing spatial data, such as photographs. However, recurrent neural networks (RNNs) are better suited for time-series analysis because they can extract temporal properties from sequential data. The processing of audio depends heavily on frequency information. Nevertheless, RNNs frequently struggle to simulate local frequency features during audio processing, a crucial aspect of audio classification applications. As a unique type of RNN, long short-term memory (LSTM) [
28] has demonstrated strong performance in long sequence applications. The LSTM network contains a significant quantity of learnable parameters, primarily utilized to mitigate the challenges of gradient vanishing and explosion that commonly arise when training lengthy sequences. Due to these benefits, LSTM networks are frequently combined with CNN networks for speech recognition [
29,
30,
31]. Although most CNN-LSTM networks can obtain better results in reclassification, most CNN-LSTM-based models suffer from highly contextual fully connected features as inputs to the LSTMs, which lack fine-detailed representations about the important features and always produce unsatisfactory recognition results [
32]. Additionally, the CNN-LSTM combination is not used as much, which underutilizes the deeper features extracted from the various layers. More seriously, the CNN-LSTM-based network model has a relatively weak attentional ability, which may lead to the interference of noise during the training process and may even lead to the degradation of the model’s classification performance [
33]. Although different attention mechanisms have been studied to try to solve this problem, there is still room for improvement regarding how to rationalize the design of attention modules corresponding to different levels of CNN features.
The remainder of the paper is organized as follows: in
Section 3, we describe the data sources and data augmentation.
Section 4 describes the methodology, network model, and attention module used in this investigation.
Section 5 focuses on the comparative results of classification accuracy on the validation set for all the models in this paper. In
Section 6, a related discussion on the identification of western black-crested gibbon calls is given. Finally,
Section 7 details the conclusions on western black-crested gibbon call recognition.
2. Related Work
Numerous techniques have been put forth in the field of chirp recognition, including network architectures like CNN, RNN, and LSTM. These techniques can be broadly divided into two categories: deep learning-based techniques and conventional feature engineering-based techniques. Conventional techniques for chirp recognition often depend on auditory feature extraction, like Mel spectrogram and MFCC. These features are fed into a machine learning classifier for classification. For example, Zhou et al. [
34] used extracted Mel spectrograms to recognize gibbon calls with a maximum accuracy of 99.8%; Rafael et al. [
35] used spectrograms to recognize 915 bird species with 71% accuracy; and Roop et al. [
36] used spectrograms to recognize bird calls with 96.1% accuracy. However, these methods have limited performance in dealing with complex audio signals and have difficulty in capturing the temporal characteristics of gibbon calls.
In 1997, Hochreiter et al. proposed the long short-term memory (LSTM) network [
28], and they are presently extensively implemented in speech processing. The problem of standard recurrent neural networks (RNNs) not being able to capture long-term dependencies can be solved by LSTM networks. Additionally, gating methods including input, output, and oblivion gates are used by LSTM networks to enhance network performance and better manage information flow. Because it can maintain and update state information at each time step, LSTM is especially well-suited for jobs involving multi-time-step predictions. This is why it is so popular in the voice recognition industry. Geng et al. [
37] has designed a speech recognition network based on LSTM network, which is mainly used to recognize the speech produced in English teaching; Ahmed et al. [
38] proposes a speech emotion recognition model based on the combination of CNN network and LSTM network; due to the inclusion of the LSTM network; the model effectively improves the overall network performance. Abdelhamid et al. [
39] presents a network architecture that integrates CNN and LSTM networks, thereby significantly enhancing the model’s efficacy in emotional speech recognition.
With the advancement of deep learning, RNN networks are increasingly employed in the speech recognition field [
40,
41]. Furthermore, the incorporation of the temporal attention mechanism significantly enhances the model’s accuracy. An example of this situation can be seen in some previous studies [
42,
43,
44], where temporal attention mechanisms have been integrated into a variety of network models to enhance the precision of speech emotion recognition. The model can learn how to adapt to changes at different time steps during the training process, because the temporal attention mechanism enables the model to dynamically assign weights to information at different time steps and to concentrate more on task-relevant time steps when processing time-series data. It enhances the network’s capacity for generalization by enabling it to adaptively simulate sequences of varying lengths.
However, as the number of network layers continues to grow, the problem of vanishing gradients occurs. In order to solve this problem, many people have conducted a lot of research, for example, He et al. [
45] proposed a deep residual structure to solve the problem of gradient vanishing in deep networks; Gustav et al. [
46] proposed a self-similarity based neural network macro-architecture design strategy to solve the problem of gradient vanishing; and Rupesh et al. [
47] proposed a highway network that allows information to flow unimpeded on a multilayer information highway.
Although existing studies have made some progress in gibbon call recognition, there are still some shortcomings. Our proposed SA_DenseNet-LSTM-Attention model achieves innovation by (1) adopting the DenseNet network as the backbone network to enhance the feature extraction capability; (2) incorporating a self-attention mechanism after each DenseNet convolutional layer to dynamically adjust the feature weights; (3) combining the LSTM and temporal attention mechanism to capture the time-dependence of audio signals and effectively extract sequence features with different chirp lengths; (4) fusing features using PCA principal component analysis to further improve the model performance. The experimental results show that our method significantly outperforms existing methods in terms of accuracy, precision, recall and F1 score.
6. Discussion
In this paper, we present a new recognition network that uses a PCA method for feature fusion to fuse the features extracted by the LSTM network based on a temporal attention mechanism and those extracted by the DenseNet network. While the LSTM network can extract temporal information from a time series, CNN focuses more on computing spatial structure, despite its superior classification capabilities. To validate the performance of our proposed model, we compared three different models, VGG16, MobileNet and Xception, and our proposed model (SA_DenseNet-LSTM-Attention) obtained the highest accuracy (
Figure 8 and
Table 7). The classification performance of the models VGG16, MobileNet and Xception also improved after adding the LSTM network based on the temporal attention mechanism (
Table 7); this result further illustrates that the LSTM network can effectively extract the sequence features into the audio. Our experiments demonstrate that our proposed SA_DenseNet-LSTM-Attention model can effectively recognize gibbon calls, which greatly reduces the time required for manual recognition, and also proves that passive acoustic monitoring combined with deep learning will be an effective tool for monitoring the western black-crested gibbon population.
Currently, the acoustic monitoring system built by project team member Zhong et al. [
7] in the Mourning Mountains allows for 24 h the monitoring of the western black-crested gibbon group. Long-term acoustic monitoring will yield hundreds of thousands of hours of recordings annually, and the manual labeling of these recordings is sometimes not feasible due to post-processing data. Our results suggest that deep learning combined with passive acoustic monitoring could be a useful approach for late-stage species-specific identification. This has the ability to reduce costs, save time and labor, and streamline monitoring processes. Our method works well for call recognition in western black-crested gibbons and may be easily extended to other call species. Accurately identifying individual vocalizing species is important for ecological sustainability.
Although our model, with a 98.2% recognition rate, is not sufficient to achieve the full recognition of western black-crested gibbon calls, the model can recognize most of the gibbon calls in the recorded files, greatly improving the efficiency of manual annotation and manual recognition of these data. With or without data augmentation, our model outperforms the other three models in terms of accuracy and generalization performance (shown in
Figure 8). This is mostly due to two factors: first, our network has a greater depth at which the gradient vanishing problem can be solved effectively; second, our network has fewer parameters yet performs better. The only negative aspect is how memory-intensive model training is. The precision of upcoming models will be enhanced by expanding the sample size of labeled data.
To achieve high recognition accuracy, our model requires a large labeled dataset; imbalances in the dataset can restrict the model’s learning ability. Therefore, expanding the dataset is a common deep learning training practice known as “data augmentation”. For example, Lasseck et al. implemented data augmentation by adding noise, filtering, or mixing audio clips [
54]. This method has been proven to effectively improve the model’s classification accuracy. Our study evaluates the impact of various speech data augmentation methods on model classification performance, finding that most traditional speech data augmentation methods do not improve and may even decrease model performance. Compared to traditional methods, datasets obtained using deep learning approaches can significantly enhance each model’s classification performance (see
Figure 9). WaveGAN employs Generative Adversarial Networks (GANs) to generate audio samples, which are more realistic and of higher quality, closer to the quality of the original audio. WaveGAN is capable of generating diverse audio samples that simulate different noises, speaking styles, and speech speeds, thus providing richer data augmentation and improving the model’s generalization ability. WaveGAN can be tailored to various speech recognition or audio processing tasks by adjusting the training approach, thereby offering broad application potential [
71,
72].
We have incorporated knowledge from several important studies that demonstrate the complexity and diversity of animal communication into our classification model for vocalizations. Following the guidance of Payne and McVay [
73], our model accommodates the repetitive patterns found in calls, which are characteristic of structured vocal sequences. Additionally, Whaling et al. [
74] provide a foundation for understanding the frequent repetitions in these sequences, which our model seeks to identify and classify. The distinctions between simple calls and more complex composites, as discussed by Behr and von Helversen [
75], and Bohn et al. [
76], have been critical in refining our approach to differentiate between monosyllabic calls and more structured composites. These studies collectively form the theoretical backbone of our research, supporting our methodology and ensuring that our classification scheme aligns with established understandings in the field of bioacoustics. Both the deep learning methods Fre-GAN and WaveGAN used in this paper can effectively improve the classification accuracy of the model, but compared to the Fre-GAN model, the WaveGAN model can achieve a better classification performance, and Fre-GAN needs more training datasets to achieve good performance. Previous studies have also demonstrated that WaveGAN can effectively improve model classification accuracy [
77,
78,
79].
To enhance the classifier’s performance in this study, we use data enhancement approaches. Nevertheless, there are certain restrictions on data augmentation. Firstly, the synthetic variants introduced by the data improvement approach might not accurately represent the intricacy of gibbon cries seen in nature. As a result, the classifier’s performance in actual applications may not match that of the enhanced dataset. In addition, the effectiveness of data enhancement techniques relies on the choice of enhancement method and parameter settings, which, if not properly selected, may introduce noise or irrelevant information, thus affecting the accuracy of the classifier. To evaluate the performance of the classifier in real-world environments, we tested it on one month of independent, non-enhanced data (validation set). According to the test results, the classifier’s performance on unenhanced data offers a more accurate assessment of how well it works in actual passive acoustic monitoring activities. We are able to obtain a more precise image of the classifier’s performance in practical applications, including its precision, recall, and general robustness, by examining the data collected during this time. To evaluate the performance of the classifier in real-world environments, we tested it on one month of independent, non-enhanced data (validation set). The test results show that the performance of the classifier on unenhanced data provides a more realistic evaluation of its effectiveness in real-world passive acoustic monitoring tasks. By analyzing the data over this period, we are able to obtain a more accurate picture of the performance of the classifier in real-world applications, including its precision, recall, and overall robustness. The test results show that the model in this paper outperforms all other models on the validation set (shown in
Table 6).
In sound classification tasks, deep convolutional neural networks outperform shallow convolutional neural networks, according to research findings [
80]. Nevertheless, as the depth of the network further expands, it encounters the challenge of vanishing gradients, and this degradation does not occur due to overfitting. Higher training errors occur if more network layers are added to a network model of appropriate depth [
81]. In order to solve the degradation problem, Huang et al. [
56] proposed a densely connected convolutional network (DenseNet).The DenseNet network is the main recognition network in this manuscript because of its excellent performance. Furthermore, we provide the LSTM network, which is predicated on the DenseNet network’s temporal attention mechanism. A recurrent neural network (RNN) variation, the long short-term memory (LSTM) model performs better than RNNs in solving gradient vanishing and gradient explosion problems, and the LSTM network shows very good model performance in the sound classification task [
82,
83]. Longer call sequences can be recognized more effectively by the LSTM network, because it addresses the issue of long-term reliance in RNNs when dealing with lengthy sequence data. On the other hand, the LSTM network may dynamically devote attention to information at different time steps in the sequence thanks to the temporal attention mechanism. As a result, the network can concentrate less on time steps that are secondary or unimportant to the current activity and more on time steps that are significant. Since the sound data used in this article has variable duration, an LSTM module based on the attention mechanism is included to help the network better generalize to long unseen sequences and be flexible in handling input sequences of varying lengths. Incorporating a temporal attention mechanism can also improve the retention of crucial information and raise attention to crucial time stages. In order to reduce computational complexity, the network can also selectively focus on time steps that are pertinent to the job thanks to the temporal attention mechanism.
Our model can classify new sound recordings with high accuracy. The predictions are almost perfect when augmented with WaveGAN speech data and when the environmental conditions used to train the model are similar. In our test dataset, the SA_DenseNet-LSTM-Attention model proposed in this paper clearly detected all gibbon calls in the test data at the cost of ten false positives. The primary cause of the false positives was that certain bird cries were identified as weakly modulated figures. Additionally, the majority of birds’ mistake rates were focused in the event of more mixed noises.