An Easily Customizable Approach for Automated Species-Specific Detection of Anuran Calls Using the European Green Toad as an Example

Abstract

Machine learning approaches for pattern recognition are increasingly popular. However, the underlying algorithms are often not open source, may require substantial data for model training, and are not geared toward specific tasks. We used open-source software to build a green toad breeding call detection algorithm that will aid in field data analysis. We provide instructions on how to reproduce our approach for other animal sounds and research questions. Our approach using 34 green toad call sequences and 166 audio files without green toad sounds had an accuracy of 0.99 when split into training (70%) and testing (30%) datasets. The final algorithm was applied to amphibian sounds newly collected by citizen scientists. Our function used three categories: “Green toad(s) detected”, “No green toad(s) detected”, and “Double check”. Ninety percent of files containing green toad calls were classified as “Green toad(s) detected”, and the remaining 10% as “Double check”. Eighty-nine percent of files not containing green toad calls were classified as “No green toad(s) detected”, and the remaining 11% as “Double check”. Hence, none of the files were classified in the wrong category. We conclude that it is feasible for researchers to build their own efficient pattern recognition algorithm.

1. Introduction

Machine learning (ML) approaches as part of artificial intelligence systems for visual and sound pattern detection are receiving increasing popularity because of their high efficiency and comparatively simple application [1,2]. For instance, machine learning algorithms have been used to detect lesions in medical images [3], leukemia in blood cell count data patterns [4], movement and behavior of livestock [5], individual pig recognition [6], as well as face and speech recognition [7,8]. In the natural sciences, some identification applications of plants and animals (e.g., iNaturalist, observation.org) are assisted by artificial intelligence [9], but the algorithms are usually not open source or freely available. Most ML algorithms were developed by hiring costly programmers and need considerable model training data for acceptable performance. Hence, for biologists with specialized (‘niche’) research questions, artificial intelligence systems such as those described above might be out of reach.

In addition, automated call detection and species identification of amphibians are less widely available compared to other taxa (e.g., birds and bats). The available tools are based on specific call characteristics such as pulse rate and frequency [10], hence they might lack specificity in cases of species with similar calls, high levels of background noise, simultaneously calling individuals, or more specific research questions [11]. Furthermore, manual or semi-automatic preprocessing steps of the audio files are often needed to improve detection performances [12]. Thus, researchers would benefit from an easily adaptable small-scale approach, where they use familiar toolsets such as software R [13] to implement ML into their data analysis.

Earlier research has shown that algorithms based on decision trees might be a promising avenue for anuran call recognition [14]. A state-of-the-art candidate machine learning algorithm is the eXtreme Gradient boosting algorithm (XGboost) [15]. It is a further development of the gradient boosting approach and has consistently performed very well in multiple competitions hosted by the Kaggle AI/ML community platform [16]. Importantly, XGboost can be applied using all standard programming languages for statistical computing used in natural sciences, including R (via the xgboost package [17]).

While many species sampling efforts in field biology are based on visual detection, some taxa can be easily identified by the sounds they produce. For instance, species-specific vocalizations can be used to monitor birds [18] and frogs [19]. Especially for citizen science, acoustic detection is already used to collect vast datasets of species distributions [20,21]. However, anuran species acoustic identification is hardly used in citizen science compared to bird or bat species [22,23,24]. Recently the automated recording of soundscapes became a promising tool for efficient acoustic monitoring, even going beyond species identification [25].

Despite the advances in digitalization and automatization, a lot of the call identification of recorded audio still requires time-consuming effort by researchers [26,27], in particular for citizen science projects that collect very large datasets [28]. In part, utilizing existing ML algorithms or creating new ones might not be feasible (e.g., high costs); furthermore, the scope of existing algorithms might not fit the needs of a particular research question.

In the current study, we show that open-source tools can be easily used to develop a highly efficient call detection algorithm, at least for a single species, using the European green toad Bufotes viridis as an example. Our study is not a comparison between different possible detection algorithms, and there might be other equally or better performing solutions. However, for our approach, we provide a step-by-step instruction to reproduce our approach, with example sound files and annotated code including explanations on how it could be customized for other species. We expect our work to be of use to biologists working with sound data, extending the available tools in wildlife research.

2. Materials and Methods

2.1. Example Species and Call Characteristics

We used the breeding call of the European green toad (B. viridis Laurenti, 1768) as an example. The green toad is widely distributed in Europe and Asia [29], however, its populations are often threatened locally [30]. Its call is a trill with a fundamental frequency of about 1200 to 1400 Hz with about 20 pulses per s, with note lengths of about 4 s [31,32]. As the calls of different Bufotes species and subspecies may vary [29], in our study we focused on the nominate subspecies B. viridis viridis and only trained and tested on calls recorded in its range.

A previous study using passive acoustic monitoring has demonstrated the need for a green toad sound detection and identification algorithm [33]. In addition, it is the target species in the ongoing citizen science project AmphiBiom (https://amphi.at; accessed on 20 August 2024), with one of its goals being to use breeding calls submitted by volunteers to identify green toad occurrences; an automated algorithm could greatly increase the efficiency of related analyses.

2.2. Data Sources

To train the ML algorithm, we obtained audio files with and without calls of green toads from public platforms, provided by colleagues, or recorded by us (see details in Supplementary Tables S1 and S2). Recordings without green toad calls included other anuran calls, crickets, birds, and background noise. Files were recorded using different methods and in different audio formats, with professional microphones or mobile phone microphones.

The field collected data to be analyzed comprised 111 sound files recorded as MP3 audio files and uploaded by citizen scientists using the dedicated mobile phone application AmphiApp (https://www.amphi.at/en/app; downloaded on 2 May 2024). Each observation was identified by two of the amphibian experts in the team (Y.V.K. and S.B.).

2.3. Data Preparation

Before analysis, all files were converted to the WAV audio format if needed using the av_audio_convert function in the package av in software R (av::av_audio_convert) [34]. To train/test the ML algorithm, we extracted the specific parts with green toad calls using the R packages tuneR and seewave [35,36]. For non-green toad calls, we randomly selected 5 min long sections of each recording for algorithm training (to reduce the computation time); for samples below 5 min in length, the entire audio recording was used. We converted the audio files to data matrices. For this, we first resampled the wave to meet our target sampling rate (44,100 Hz). Then we filtered the frequency range between 1150 and 1650 Hz (function seewave::ffilter) and created a spectrogram object function seewave::spectro). For generating the spectrogram, we used the default parameters including the 512 window length, which resulted in a time difference between two time points of 0.0116 s. The amplitude values were then extracted and transposed, resulting in a matrix with frequency as columns and time as rows. To also represent previous and following frequency patterns in each time point, we added the row values of the 18 previous and 18 following rows to each row as additional columns (representing +/−0.2 s), hence, increasing the number of columns in the data set. As the columns were always arranged in the same way, each time point had the information about the amplitude per frequency for the current time point as well as the 18 previous and 18 following time points. Therefore, the algorithm has the possibility to be trained to the combination of amplitudes per frequency along a time period of about 0.4 s. The resulting matrix was then duplicated and scaled (function base::scale). Hence, the final matrix consisted of scaled and raw amplitude values for each row (time point). In addition, only for algorithm training and testing, we added a column indicating if the data are from a green toad call or not.

2.4. Call Detection Algorithm Training and Testing

We used the extreme gradient boost machine learning algorithm (function xgboost in the package xgboost [17]) for our green toad detection approach. Throughout we used a maximum tree depth (max.depth) of 100, an eta (i.e., learning rate) of 1 and 100 boosting rounds (nrounds); for all other parameters, we used the default values. The learning task of the algorithm (objective) was defined as a logistic regression for binary classification (binary:logistic).

First, we trained the algorithm on randomly selected 70% of the data (training data) and then compared the performance on the remaining 30% (test data). We used the confusionMatrix function of the caret package [37] to test the performance of the xgboost predictions (function predict) on the test data.

Then we combined all data (the training and testing data from above) and repeated the algorithm training steps to make use of all the available data. The resulting algorithm was then applied on the field collected data, returning an assessment for each file (“Green toad(s) detected” if ≥10 consecutive time points are identified as containing a green toad call, “No green toad(s) detected” if less than two consecutive time points contain green toad calls, or “Double check” otherwise).

Following the application of the algorithm, we did a blinded manual identification of the audio files to identify files with and without green toad calls.

2.5. Example Protocol for Customized Call Recognition

For the example protocol to build customized detection algorithms for calls, we provided a suggestion for folder structure, an overview script (0_overview.R) that calls the scripts needed for data preparation (1_global_constants.R to centralize setup, 2_convert_audio_to_WAV.R, 3_target_species_sound_cleaning.R, 4_create_sound_matrices.R); a file for algorithm training (5_MLA_training.R), and the machine learning prediction function together with customized helper functions (functions_sound_identification.R). We also shared an application for this function using sound files not used for training. We worked through our example ensuring an easy workflow and general acceptable performance of this small-scale example. In the comments of the script together with the information in a help file (“Compiling a machine learning algorithm to detect target animal sounds.docx”), we provided suggestions for ways to adapt the algorithm for other sounds (files can be found in S1: Call_Example_Procedure.zip).

3. Results

We developed a highly efficient machine learning call detection algorithm, at least for a single species that can easily be adapted to other study cases. These results were obtained even if the field data audio recordings varied significantly, both in terms of what was recorded (e.g., no amphibians at all) and in the audio quality (e.g., including background noise and echo-like artifacts).

3.1. Call Recognition Performance

Within the training and testing datasets, the detection algorithm had an accuracy of 0.9948 (95% CI: 0.9946–0.9951). After training the algorithm on the combined data set (i.e., testing data set combined with training data set), we tested the resulting detection function (function: green_detection in Supplementary File: functions_calls_recogn_Aug2024.R; for the most up-to-date function and algorithm please visit the following GitHub page: https://github.com/lukasla/Amph_sounds) on 111 files downloaded from the AmphiApp back-end on 31 May 2024. Of 111 files to be analyzed, we removed 12 as they were <2 s long or empty; 29 files contained B. viridis calls (see example Figure 1A). The remaining 70 files contained no green toad calls. Of the 29 audios containing green toad calls, 90% (n = 26) were correctly classified as “Green toad(s) detected”, and 3 as “Double check”, with no misidentifications. Of the 70 files not containing green toad calls, 89% (n = 62) were correctly identified as “No green toad(s) detected”, and 8 as “Double check”. When only considering two categories, green toads present or not present, the detection function missed 3 out of 29 (i.e., 10%) green toad sound files (e.g., because of loud background noise, see Figure 1B). No files that did not contain green toads where misclassified. When double-checking files classified as “Double check”, the success rate would be 100% in both cases and it would reduce the workload by a factor of about 10 (i.e., checking 11 files instead of 99). The application of the algorithm did not require highly specialized or difficult to access computing power. We carried the calculations using a high-end personal computer (memory: Kingston 128 GB DDR4 3200 MHz, processor: AMD Ryzen Threadripper Pro 24 cores 3.80 GHz, GPU: AMD Radeon PRO W6400 4 GB).

3.2. Example Protocol for Customized Call Recognition Algorithm

We provide an easy-to-use procedure to implement your own species-specific call detection algorithm using freely available packages in R (Supplementary Zip File: Call_Example_Procedure.zip). Possibilities to tweak the algorithm, especially regarding the preparations of the sound file information are described in the Supplementary Materials. The R code and sound files we provide are restricted to some AmphiBiom sound files and a few additional files (see list in the working example folders) therefore do not encompass the database we used for the detection algorithm. Hence, one should not use the obtained “example algorithm” as a substitute for the green toad detection function described above. However, we provide these files to show the stepwise procedure to build the algorithm, which should enable other researchers to establish their own detection algorithm. We provide eight test files from our AmphiBiom project, not used in model training, four of which contain green toad calls. The resulting algorithm used inside our target species detection function identified all four green toad files correctly, the remaining files were either classified as “Double check” or “Target species not detected”.

4. Discussion

We demonstrate that customized, semi-automated, and highly accurate anuran call detection algorithms can be easily built and implemented using open-source software, if there are sufficient suitable reference vocalizations available. Our example protocol can be utilized by biologists of all areas working with sounds to build their own customized detection algorithm with little effort. Furthermore, we provide the tools in an environment (R) that should be familiar to a wide swath of researchers, running on major operating systems (Windows, Mac, Linux), and does not depend on external software companies or specialized programmers. However, like detection algorithms for bird songs (for example, the Merlin Bird ID application by The Cornell Lab of Ornithology), such tools may be integrated with available citizen science mobile phone applications such as iNaturalist or observation.org.

We show that the detection algorithm we built was robust and specific. It detected green toad calls when other amphibians were present, even with high background noise and low sound quality. However, great care should be taken in preparing the training and testing data sets to include sounds that are like the target species vocalizations, as well as common background noise. The best results are expected to be obtained when the training data cover the range of possible sound files to be evaluated by the algorithm. Hence, for a citizen science application, it would make sense to use the initial data collection period to obtain a training data set and then continue within the same framework but add the algorithm to speed up call identification. The size of the data set required likely varies a lot between research questions, target sound type, and file qualities.

Importantly, we prepared the algorithm to detect one specific call, the green toad breeding call. In cases of animals with several sound types (e.g., mating, warning, territorial calls), it might be possible to include all of them in one algorithm training procedure. However, in this case, the different calls could not be separated afterwards other than manually. Therefore, we would suggest training as many algorithms as there are sounds that should be detected and include all of them in one detection function. In this instance, one could differentiate not only if their target species is present, but also their behavioral state. The advantage of training several algorithms is that they can detect different sound types (e.g., warning versus mating) even when they are happening at the same time point. Similarly, one could train several algorithms for several different species and detect all of them in a single audio file, if all the algorithms are included in the detection function. Moreover, animal sounds can change from younger to older [38] and smaller to larger individuals [39] of the same species, potentially enabling researchers to estimate age/size structure through audio surveys.

We advise against training the data set on calls from one specific region and limited time frame in the season and then using the algorithms in a different area or/and different seasons. Small variations in call characteristics, but also background/non-target sounds between populations or regions (e.g., [40]) that could be missed by human ears, might decrease the accuracy of algorithms trained and applied on sound files obtained from different locations. We noticed during testing that the training files for such algorithms should be within the range of files that you expect to obtain. For instance, training the algorithm on files of natural sounds but then testing files containing music (e.g., piano) will likely lead to false results.

Importantly, the majority of the sounds we analyzed were likely exclusively being collected with built-in smartphone microphones through the AmphiApp and directly saved in MP3 format. Thus, the sound quality might vary based on the specific hardware and software characteristics of individual devices and their ability to record and convert sounds in real time, the distance from the recording device to the calling toad, and/or background noise [22,41,42]. Furthermore, the MP3 format is widely used for audio recording as it generates relatively small-size files but achieves this by using lossy data compression consisting of inexact approximations and the partial discarding of data (including reducing the accuracy of certain components of sound that are beyond the hearing capabilities of most humans). Therefore, care needs to be taken that key characteristics of the animal calls recorded and analyzed are not lost during recording/conversion and subsequently unavailable for analysis and recognition by the algorithm. Training data should thus include a range of sounds with varying recording quality. However, considering these limitations, at least in our case, the algorithm was robust and accurate even when analyzing recordings with quality deemed poor to the human listeners.

Using machine learning approaches for automated species detection in addition to professional (human) assessments could improve data quality for auditory surveys [43,44]. Such auditory surveys are common in amphibian ecology [45] and can help monitor species distribution and to some extent even population trends. New developments in automated sound recording (e.g., soundscape recordings) could easily boost our understanding of the activity and distribution of calling amphibians, as it is already transforming marine acoustic studies [46]. As most anuran calls are specifically related to reproductive efforts [47], acoustic information is highly valuable, because it relates to a reproductively active population. In the future, such calls might also be used to assess the status of populations, as the calls are related to the size [39] and potentially health status of individuals.