Singing Voice Detection in Electronic Music with a Long-Term Recurrent Convolutional Network

Abstract

Singing Voice Detection (SVD) is a classification task that determines whether there is a singing voice in a given audio segment. While current systems produce high-quality results on this task, the reported experiments are usually limited to popular music. A Long-Term Recurrent Convolutional Network (LRCN) was adapted to detect vocals in a new dataset of electronic music to evaluate its performance in a different music genre and compare its results against those in other state-of-the-art experiments in pop music to prove its effectiveness across a different genre. Experiments on two datasets studied the impacts of different audio features and block size on LRCN temporal relationship learning, and the benefits of preprocessing on performance, and the results generate a benchmark to evaluate electronic music and its intricacies.

1. Introduction

The singing voice is one of the essential components of music, as it communicates the emotions and motives of any song through lyrics. Its high level of expressiveness has made the human voice a topic of numerous study fields, one of them being Music Information Retrieval (MIR). In it, Singing Voice Detection (SVD) is the ability to identify segments of singing activity. It is one of the most researched topics due to its contributions as a preprocessing step in the performance of other tasks and applications such as Singing Voice Separation (SVS) [1,2], Vocal Melody Extraction (VME) [3], and Lyric Alignment [4,5], Lyric Transcription [6], among others.

Traditional methods combine attributes of speech with classification algorithms to detect singing voice segments in songs. For example, features such as Mel-Frequency Cepstral Coefficients (MFCC), Perceptual Linear Prediction (PLP), and Linear Prediction Cepstral Coefficients (LPCC), which are obtained by introducing cepstrum coefficients in Linear Predictive Coding (LPC) [7], and classifiers such as Gaussian Mixture Models (GMM), Hidden Markov Models (HMM) and Support Vector Machines (SVM) were introduced into speech recognition systems [8]. Rocamora and Herrera [9] studied the accuracy of SVD of various statistical descriptors and features, concluding that MFCC was the most appropriate feature, and SVMs was the best-performing classifier. However, they have certain limitations for SVD, as speaking and singing voice are different in terms of how singing uses breath and muscle tension to control pitch and duration, therefore giving it a higher average intensity and greater dynamic range compared to speech [10].

On the other hand, Deep Learning (DL) techniques have been recently used in SVD due to its feature representation and time-space modeling capabilities [11]. For example, Schlüter et al. [12] introduced a model for SVD using a Convolutional Neural Network (CNN) that could work with Mel-spectograms through data augmentation. You et al. [13] combined CNNs with MFCCs and Fast Fourier Transform (FFT) features, whereas Huang et al. [14] did it with Discrete Fourier Transform (DFT) coefficients, obtaining higher detection accuracy than MFCCs. Hughes et al. [15] proved that a Recurrent Neural Network (RNN) outperformed GMM systems on voice detection activities, and Lehner et al. [16] increased its effectiveness with a Long Short-Term Memory (LSTM) network to detect vocal and nonvocal frames.

Further attempts have mostly consisted of combining DL techniques to separate tasks and increase their performance. Leglaive et al. [17] combined a bidirectional RNN with an LSTM (BLSTM-RNNs) and used a temporal context in order to classify the input vector, whereas Cho et al. [18] used a Gated Recurrent Unit (GRU) to allow each unit to adaptively capture dependencies across time scales. Most recently, Zhang et al. [11] achieved state-of-the-art performance using SVS as a preprocessing method for classifying input vocal signals and a Long Short-Term Recurrent Convolutional Network (LRCN) to extract and sequence the vocal background in the time domain for vocal detection.

The majority of SVD systems are designed and evaluated with popular music. However, Scholz et al. [19] showed that these tend to not generalize well to other genres unseen during training, implying that focusing solely on pop music limits their general applicability. There have been attempts to train SVD models with other music genres. Krause et al. [20] proved the generalization capabilities and robustness of two models with opera recordings as a dataset. However, there are still many genres with their own peculiarities that have not been explored yet, such as electronic music.

Electronic music has been studied in music generation. Conklin et al. [21] developed a Digital Audio Workstation (DAW) plug-in to generate chord sequences and define patterns in EDM through machine learning and model sampling. However, as electronic music focuses more on repetition and looping rather than singing, it has not been explored in other study fields such as MIR. Furthermore, it is evolving into various sub-genres such as dubstep, house, techno, glitch, drum and bass, among others, each one of them with their own intricacies and techniques, so it is still a fresh and developing study area to explore.

Particular differences exists between popular and electronic music due to the singing techniques and instrumentation involved. Within electronic music, songs with a heavy emphasis on digital instruments such as synthesizers and samplers, and the usage of tools such as auto-tune, an audio processor that measures and applies subtle corrections to the pitch of vocal and instrument recordings, provide interesting opportunities to test the effectiveness of MIR algorithms in new scenarios. Additionally, previous research has shown that SVD systems may overfit to factors such as loudness [22] and singing style [19], and therefore it is hard to expect them to generalize to unseen genres.

In this paper, a LRCN is evaluated with a new dataset of electronic music to compare its performance against public-domain datasets of pop music so as to prove the effectiveness of the model in a different and unstudied genre. This will provide a baseline result to study the electronic music genre in the SVD field, and it will yield new insights into the aspects of data training when using SVD algorithms to test unexplored music genres.

This paper is organized as follows: Section 2 gives a general introduction to the SVD system and datasets used. Section 3 presents the performed experiments and obtained results on the system. Conclusions are described in Section 4.

2. Materials and Methods

2.1. Architecture Layout

The proposed system has a preprocessing step to filter the audio from the musical accompaniment to increase the performance of the feature extraction and LRCN training with the datasets, thus maximizing the results to perform the comparisons between them. The architecture of the system is shown in Figure 1.

2.2. Singing Voice Separation

Singing Voice Separation (SVS) is the task of filtering out musical accompaniment from vocals in a song to obtain a relatively pure human voice. Therefore, it complements SVD in the goal of detecting a singing voice. From all SVS algorithms, Cohen-Hadria et al. [23] concluded that the most effective method is based on U-Net, a deep neural network proposed by Jansson et al. [24] with six convolutional and six deconvolutional layers that takes the spectrum of a singing voice as input. However, if there is a string accompaniment, then the audio cannot be cleanly split, as the separated vocal signal may contain collateral noise as a result.

2.3. Feature Extraction

With the LRCN, audio features can be combined from consecutive audio frames to differentiate between human and nonhuman voices by forming two-dimensional figures in the (x,y) form of (audio frame in time, coefficients of the features). For the identification of vocal signals, the following features are used:

• ZCR (Zero-Crossing Rate): How many times a signal value changes sign (negative or positive) in a time framE.
• MFCC (Mel-Spectogram Cepstral Coefficients) [25]: Coefficients that represent the power spectrum speech based on human audition perception. First-order and second-order difference of MFCC are also computed.
• LPCC (Linear Prediction Cepstral Coefficients) [26]: Cepstrum coefficients in Linear Predicitive Coding (LPC) parameters that represent the nature of the sound based on the vocal tract shape.
• Chroma [27]: Descriptor that represents the tonal content of an audio signal.
• SSF (Spectral Statistical Features) [28]: Frequency information of the audio signal (RMSE, centroid, roll-off, flatness, bandwidth, contrast, polly).
• PLP (Perceptual Linear Prediction) [29]: Alternative to MFCC that wraps the spectra to maintain the content of speech.
• Spectrum: Representation of sound in terms of frequency vibration. It is calculated to verify the performance of the LRCN and its feature fusion capabilities.

2.4. LRCN Components

The LRCN is a deep model that learns time-series relationships and integrates information in space for deep feature extraction. The CNN layer spatially extracts the audio features for deep feature extraction, while the LSTM layer performs a deep feature encoding of the convolutional layer output to learn the temporal relationship and obtain a final singing or no-singing label.

The LRCN has an input layer with the size of the combined feature vector, an output layer with a sigmoid unit and three hidden layers, and outputs values for each frame block between 0 (no singing voice) and 1 (singing voice). By combining a feature extraction CNN and a LSTM, the model passes the fusion features of successive audio frames into a new vector and then synthesizes its temporal dynamics in a sequential order. The LRCN layers and main functions are shown in Figure 2.

The LRCN layer has a stack of one-dimensional inputs and a set of one-dimensional kernels. The max-pooling layer subsamples the inputs by taking the maximum over groups of contiguous frames, and the input layer receives the fusion features of continuous frames in a fixed block size. The parameters for the LRCN layers are shown in

Table 1. Parameters of the LRCN layers.

Layer	Output Shape	Parameter
reshape-1 (Reshape)	(None,1,1,20,276)	0
conv_lstm_2d-1 (ConvLSTM2D)	(None,1,17,13)	60,164
dropout-1 (Dropout)	(None,1,17,13)	0
max_pooling_2d-1 (MaxPooling2D)	(None,1,8,13)	0
dropout-2 (Dropout)	(None,1,8,13)	0
flatten-1 (Flatten)	(None,104)	0
dense-1 (Dense)	(None,200)	21,000
dropout-3 (Dropout)	(None,200)	0
dense-2 (Dense)	(None,50)	10,050
dropout-4 (Dropout)	(None,50)	0
dense-3 (Dense)	(None,1)	51

.

The input data are concatenated on continuous frames. The input vector is mapped to one label based on the ground truth. In the LRCN layer, 256 convolution filters and a convolution kernel operate a one-dimensional convolution along with the features, and it determines the cell state in the input grid and past states of its neighbors through a convolution operator in state-to-state and input-to-state transitions. The structure of the LRCN is shown in Figure 3. The equations are shown in Equations (1)–(5). All inputs, gates, hidden states, and cell outputs of the LRCN are 3D tensors.

$$i\left(x\right)=\sigma (W_i\ast [conv\left(X\left(x\right)\right),H(x-1),C(x-1)]+b_i)$$

(1)

$$o\left(x\right)=\sigma (W_o\ast [conv\left(X\left(x\right)\right),H(x-1),C\left(x\right)]+b_o)$$

(2)

$$f\left(x\right)=\sigma (W_f\ast [conv\left(X\left(x\right)\right),H(x-1),C(x-1)]+b_f)$$

(3)

$$C\left(x\right)=f\left(x\right)\ast C(x-1)+i\left(x\right)\ast tanh(W_c\ast [conv\left(X\left(x\right)\right),H(x-1)]+b_c)$$

(4)

$$H\left(x\right)=o\left(x\right)\ast tanh\left(C\right(x\left)\right)$$

(5)

where:

• $i\left(x\right)$ = Input gate;
• $o\left(x\right)$ = Output gate;
• $f\left(x\right)$ = Forget gate;
• $C\left(x\right)$ = Input LRCN cell;
• $H\left(x\right)$ = Output LRCN cell;
• W = Matrix of weights;
• b = Vector of bias;
• $\sigma$ = Sigmoid function;
• * = Element-wise product;
• conv. = Convolution operator.

2.5. Existing Dataset—Jamendo Corpus

The Jamendo Corpus is a public dataset of 93 copyright-free pop songs introduced by Mathieu Ramona et al. in [30]. Each song is encoded in 44.1 kHZ stereo .ogg or .mp3 format, and has a file with vocal annotations of signing and non-singing parts by the same person to provide ground truth data. The dataset is divided into a training set of 61 songs, validation set of 16 songs, and test set of 16 songs.

2.6. Proposed Dataset—Electrobyte

Since there are no datasets of electronic music for SVD purposes, a new one was compiled from songs within the genre that have singing or lyrics. In order to avoid using copyrighted music, the songs were extracted from the following sources:

• The Arcadium [31]: Record Label by German artist and producer TheFatRat dedicated to making available copyright-free gaming music.
• NCS [32]: Record Label by British youtuber Billy Woodford dedicated to making available royalty-free electronic dance music for gaming and content creator communities.

The proposed dataset, named Electrobyte, consists of 90 electronic songs from various sub-genres such as Glitch Hop, Bass, Drum and Bass, House, Dubstep, Drumstep, Chill, EDM, and Electro Pop, among others, and follows the standards of SVD datasets such as Jamendo Corpus and RWC Popular Music:

• All songs are stored in .mp3 format;
• Voice activation annotations were manually performed and stored for each song in a .lab file;
• The dataset is divided into a training set of 60 songs, validation set of 15 songs, and test set of 15 songs, which were all randomly selected for each set.

2.7. Evaluation Metrics

To evaluate and compare the results of the datasets, model predictions were compared with the ground truth labels to obtain the number of results over all songs in the test set. Accuracy, precision, recall, and F1-measure were calculated to summarize results. The metrics are defined as follows:

$$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$$

(6)

$$Precision=\frac{TP}{TP+FP}$$

(7)

$$Recall=\frac{TP}{TP+FN}$$

(8)

$$F1=2\ast \frac{Precision\ast Recall}{Precision+Recall}$$

(9)

where:

• $TP$ = Total of true positive matches;
• $FP$ = Total of false positive matches;
• $TN$ = Total of true negative matches;
• $FN$ = Total of false negative matches;
• $F1$ = F1-measure.

Accuracy is the ratio of frames classified correctly. Precision is the ratio of frames classified as voiced. Recall is the ratio of frames classified correctly as voiced. F1-measure combines precision and recall to measure the accuracy of a model on a dataset. In this case, F1-measure is the most important metric, as it implies how correct our model may be when classifying the frame of a song.

3. Experiments

3.1. Technical Requirements

Four experiments were conducted in total. The first involved comparing the performance of the audio features and their combinations as input to the proposed LRCN to find the optimal feature fusion. In the second experiment, the block size of continuous frames for the input layer was tested. In the third, SVS was evaluated to verify its effectiveness as a preprocessing step. In the last experiment, a comparison between the tested datasets and the state-of-the-art SVD systems was conducted. To replicate them, the proposed LRCN can be found in the following Github repository: https://github.com/Raylogic/SVD-LRCN (accessed on 9 June 2022). The results provided are the typical values obtained after several iterations of the experiments. The general parameters and configuration values used were the following:

• For the training of the LRCN, the following parameters were used across all experiments, based on the model proposed by Zhang et al. [11]:

  –

  Batch size = 32;

  –

  Drop-out rate = 0.2;

  –

  Learning rate = 0.0001;

  –

  Number of epochs = 10,000;

  –

  Early stopping = true.

• Feature vectors are calculated by dividing the audio signal into overlapping frames and calculating the FFT (Fast Fourier Transform) of each frame in a Hamming window. After computing and combining the coefficients of MFCC, LPCC, PLP, chroma, and SSF, each feature vector contains 288 coefficients, which then are used to classify the vocal or nonvocal segments with the LRCN model. The frame parameters are based on the values proposed by Zhang et al. [11]:

  –

  Sampling rate = 16,000 Hz;

  –

  Low-frequency limit = 0 Hz;

  –

  High-frequency limit = 800 Hz;

  –

  Overlap size = 1536 samples (0.96 s);

  –

  Frame size = 2048 samples (1.28 s).

• For the hardware requirements, the system was operated and trained on a laptop with an Intel(R) Core(TM) i7-9750H CPU @2.60GHz and 16GB of RAM.
• For the software requirements, Python 3.9.5 was used to develop the SVD system. Additionally, the following Python libraries were installed:

  –

  audioread—2.1.9;

  –

  chainer—7.8.1;

  –

  ffmpeg—1.4;

  –

  h5py—3.6.0;

  –

  joblib—1.1.0;

  –

  keras—2.8.0;

  –

  librosa—0.9.1;

  –

  matplotlib—3.4.2;

  –

  numba—0.55.1;

  –

  numpy—1.20.3;

  –

  pandas—1.4.1;

  –

  playsound—1.2.2;

  –

  pydub—0.25.1;

  –

  reportlab—3.6.9;

  –

  scikit-learn—1.0.2;

  –

  scipy—1.8.0;

  –

  SoundFile—0.10.3;

  –

  spectrum—0.8.1;

  –

  tensorflow—2.8.0;

  –

  tqdm—4.64.0;

  –

  wavfile—3.1.1.

To perform the following experiments, preprocessing steps were carried out beforehand on each dataset with the Preprocessing.py script. First, songs were transformed into .wav format. Then, SVS was performed on all of them through U-Net, generating two audio files per each (raw and vocal audio signal). Feature extraction was then performed on the audio files and stored in a .joblib file, where the feature vectors were generated per block and its associated sing/nosing label was added. Finally, the .joblib files were compiled in two .h5 files (raw and vocal audio signal), where the data were separated into training, valid and test sets. The files were later used to train the LRCN for the SVD task.

In the LRCN.py script, LRCN layers and tasks were configured, which specifically described the necessary functions to extract the data form the .h5 files, train and validate the model, load model weights, perform Early Stopping, predict the presence or absence of singing on the test set songs and obtain the evaluation metrics by calling the Evaluation.py script. In addition, it contained the code to perform the four experiments and store their results on CSV files. The experiments on the datasets were called from the SVD.py script, while the plots from the results were generated on the Box.py script.

3.2. Performance across Features

In the first experiment, features were individually selected to perform the SVD task on the separated vocal signal of the Jamendo and Electrobyte datasets. Based on their F1-measure performance, they were selected or dropped for the next phase of the experiment. To determine the best feature combination, based of their performance of a single feature, each was added into the combined feature vector one by one. Once all of the best combinations were evaluated, the one with the best F1-measure was selected as the input for the LRCN.

The performance of the features on the separated vocal signal evaluated individually on the Jamendo dataset is shown in Figure 4, and on the Electrobyte dataset in Figure 5.

From the results in Figure 4 and Figure 5, the chroma feature exhibited the poorest performance for SVD task, while the SSF achieved the best one. The feature sets were concatenated in each frame and extracted with the convolution layer of the LRCN. For the feature combination phase, the first feature added to the vector was the SSF given its highest F1-measure, and the rest were incrementally added to evaluate all possible combinations. The results of the combined features are shown in Figure 6 for the Jamendo dataset and Figure 7 for the Electrobyte dataset.

From the results on the Jamendo dataset shown in Figure 6, SSF, MFCC, LPCC, and PLP was the feature combination that achieved the best performance. For the Electrobyte dataset in Figure 7, the best feature combination was SSF and PLP. With the observations from both datasets, the combined features were set as the SSF and PLP.

3.3. Block Size Setting

In the second experiment, the input frame-block size was iterated to find the optimal number of successive frames to feed the LRCN input layer and serve as a control variable for the datasets (Jamendo and Electrobyte). Block sizes from 5 to 29 frames were evaluated to find the amount that gave the best F1-measure and most consistent metrics. Figure 8 shows the block sizes performance on the Jamendo dataset, and Figure 9 on the Electrobyte dataset.

From the results on the Jamendo dataset, the effect of block size in the LRCN barely fluctuated in terms of F1-measure, whereas on the Electrobyte dataset the differences were a little more pronounced. Based on the experiments and the research performed by Zhang et al. [11], the input block size for the LRCN was set to 20 frames.

3.4. Effects of Singing Voice Separation

In this implementation, SVS was adopted as a preprocessing step for SVD through the U-Net network to separate vocals from musical accompaniment and generate two versions of audio signals (raw and audio vocals). The third experiment consisted of extracting the features of both versions in both datasets to train the LRCN and validate the effects of the preprocessing by comparing its F1-measure. The model was set with the block size of 20 frames set on the previous experiment, and a vector of 276 components to remove the feature of chroma (12). The results were compared with both the split vocal and raw audio. Performances are shown in Figure 10.

From the comparison of the classification results from both datasets, the use of separated vocal signals exceeded the raw signals in terms of F1-measure, so the application of SVS as a preprocessing step does improved SVD performance. However, it is noticeable that the difference was way more pronounced in Jamendo (more than 20%), whereas in Electrobyte it was minimal (less than 2%). This contrast of results between datasets can be attributed to the unique characteristics of electronic music in terms of voice management. However, further research needs to be performed to pinpoint the exact causes of the result disparities.

3.5. Comparison with Related Works on Existing Datasets

As the last experiment, to validate the proposed SVD system, both datasets were used to perform 10 prediction results. From all of the evaluation metrics, the F1-measure provides the most information about the effectiveness of the model. Furthermore, the standard deviation was calculated to validate the consistency of the results. The model was set with the best parameters obtained on the previous experiments (vector of 276 components, block size of 20 frames, SVS as a preprocessing step). The averaged results are shown in

Table 2. Performance of the SVD system on two different datasets.

	Accuracy	Precision	Recall	F1-Measure	Deviation
Jamendo	0.939	0.937	0.945	0.942	0.004
Electrobyte	0.833	0.798	0.861	0.828	0.006

.

The SVD system performed very well on the Jamendo dataset, with an F1-measure of 0.942 and standard deviation of 0.004, whereas on the Electrobyte dataset it achieved an F1-measure of 0.828 and standard deviation of 0.006. To validate the robustness of the model, a K-fold cross-validation was performed on both datasets, in which the training, validation, and test sets were resampled seven times by randomly selecting the songs for each. Across all seven samples individually, the standard deviation for the averaged F1-measure on Jamendo was lower than 0.004, whereas for Electrobyte it was lower than 0.008. It is worth mentioning that by repeating the experiment several times, one of the samples performed better than the original set on both datasets, with an F1-measure of 0.966 on Jamendo and 0.885 on Electrobyte. However, when considering the results of the samples and the original dataset as a whole, the standard deviation for the averaged F1-measure on the Jamendo dataset was 0.016, whereas for the Electrobyte dataset it was 0.021. Although the performance of the system in Electrobyte was lower compared to Jamendo across all samples, as this is the first study evaluating SVD in electronic music, these results will serve as a framework for future studies about the intricacies of the genre and how to improve the detection of vocals with more effective models.

Finally, the SVD system was compared with the best value performance from Ramona [30], Schlüter [12], Lehner-1 [33], Lehner-2 [34], Lehner-3 [16], Leglaive [17], and Zhang [11] on the Jamendo dataset. The comparison results are shown in

Table 3. Comparison of the SVD system with existing methods on Jamendo.

	Accuracy	Precision	Recall	F-Measure	Deviation
Ramona [30]	0.822	-	-	0.831	-
Schlüter [12]	0.923	-	0.903	-	-
Lehner-1 [33]	0.882	0.880	0.862	0.871	-
Lehner-2 [34]	0.848	-	-	0.846	-
Lehner-3 [16]	0.894	0.898	0.906	0.902	-
Leglaive [17]	0.915	0.895	0.926	0.910	-
Zhang [11]	0.924	0.926	0.924	0.927	-
LRCN	0.939	0.937	0.945	0.942	0.004

.

Table 3 shows the comparison of the results of the proposed LRCN on the Jamendo dataset against those obtained on Jamendo in related reports under the same conditions. When compared with the classifiers based on LSTM, the proposed LRCN achieved an F1-measure of 0.942, which outperforms the BLSTM of Leglaive and LSTM of Lehner-3 by 4%. Furthermore, it shows an improvement over the state-of-the-art method reached by the LRCN of Zhang by 1.5%.

4. Conclusions

In this paper, an SVD system based on LRCN was presented to detect vocals in a new electronic music dataset and evaluated against previous DL models (CNN, LSTM, LRCN) using existing pop music datasets, exhibiting an F1-measure of 0.828 for the electronic music dataset (Electrobyte); however, this was lower compared to its pop music genre counterpart (Jamendo) with an F1-measure of 0.942. This provides a benchmark result to further evaluate electronic music in the SVD field with new models and achieve a better performance.

Future work will expand the research on electronic music attributes to find the root causes of the result in terms of vocal signals. Furthermore, new datasets for different genres as popular as pop music (such as rock or hip-hop) will be created to evaluate the LRCN and perform the same experiments. Finally, datasets with combinations of various music genres will be generated and tested to prove the robustness of SVD systems and find patterns or similarities between genres.