Pyramid Feature Attention Network for Speech Resampling Detection

Abstract

Speech forgery and tampering, increasingly facilitated by advanced audio editing software, pose significant threats to the integrity and privacy of digital speech avatars. Speech resampling is a post-processing operation of various speech-tampering means, and the forensic detection of speech resampling is of great significance. For speech resampling detection, most of the previous works used traditional methods of feature extraction and classification to distinguish original speech from forged speech. In view of the powerful ability of deep learning to extract features, this paper converts the speech signal into a spectrogram with time-frequency characteristics, and uses the feature pyramid network (FPN) with the Squeeze and Excitation (SE) attention mechanism to learn speech resampling features. The proposed method combines the low-level location information and the high-level semantic information, which dramatically improves the detection performance of speech resampling. Experiments were carried out on a resampling corpus made on the basis of the TIMIT dataset. The results indicate that the proposed method significantly improved the detection accuracy of various resampled speech. For the tampered speech with a resampling factor of 0.9, the detection accuracy is increased by nearly 20%. In addition, the robustness test demonstrates that the proposed model has strong resistance to MP3 compression, and the overall performance is better than the existing methods.

1. Introduction

In the rapidly evolving field of digital speech, particularly in the context of digital avatars, the ease of audio editing and modification through software (such as Audacity 3.4.2 [1], developed by Audacity Team, sourced from Carnegie Mellon University, Pittsburgh, USA; Adobe Audition 23.5 [2], developed by Adobe Systems, sourced from San Jose, California, USA; and GoldWave 6.80 [3], developed by GoldWave Inc., sourced from St. John’s, Newfoundland and Labrador, Canada) raises critical concerns for security and privacy. The undetectable manipulation of digital speech avatars poses substantial threats to judicial processes, political fields, and social security. Contemporary speech forensics techniques are pivotal in ensuring the integrity of digital avatars and focus on detecting tampering facilitated by audio editing software, such as deletion, insertion, copy and move, splicing, resampling and recompression of audio clips [4,5,6,7]. It is worth noting that in the field of speech content forensics, there are more forensic methods for speech deletion, copy and move, splicing, and other tampering approaches [8,9,10], while there are relatively few methods for speech resampling forensics, and these tampering means are often accompanied by resampling operations. Thus, the study of speech resampling detection has significance in complex forensics environments, which will provide comprehensive tools and methods for forensics to ensure the authenticity and integrity of audio.

Speech resampling refers to converting the original sampling frequency of speech into a new sampling frequency. The bulk of the existing detection methods for speech resampling are inspired by detection algorithms of digital image resampling [11,12]. According to the fact that the resampling process will introduce periodic information into the original speech, Yao et al. [13] proposed an audio resampling detection method based on expectation maximization (EM), which uses the spectral statistical moments of fixed sample points as input features. At the same time, the audio amplitude histogram is introduced to approximate the audio signal distribution more accurately. However, this algorithm can only detect resampled speech under linear interpolation. For this reason, Chen et al. [14] improved the algorithm of Yao et al. [13]. The EM algorithm was used to simultaneously estimate a set of periodic sample points with domain correlation and specific correlation coefficients, and obtain the normalized probability vector of all sample points to observe whether it exhibits periodicity. The method is able to detect resampled speech using different interpolation methods. The algorithms in [13,14] require highly accurate selection of initial parameters to reduce the impact of noise. Factors such as noise and signal distortion can lead to lower detection accuracy in speech down-sampling. Shi et al. [15] detected resampling by analyzing the interpolation characteristics of resampled speech and studying the singular values of the signal formed by the sampling points. Specifically, after the singular value decomposition of the speech signal, a set of singular values and the corresponding left and right singular signals are obtained, and the variation in the speech signal is indicated by counting the average number of zero singular values. Hou et al. [16], referred to the image forensics method proposed by Farid et al. [17], that an image would show periodic domain correlation after resampling. They found that after resampling an audio signal through the second-order difference, the interval between its zero values is periodic, and this feature was applied to nonlinear interpolation. However, different thresholds are selected by this method for different resampling factors and interpolation kernels, so it has limitations in practical application. Wang et al. [18] calculated the logarithm ratio of band energy between original and resampled speech according to the characteristics that the bandwidth of the resampled speech is inconsistent with its sampling rate, so as to distinguish true and false speech. The algorithm can detect resampled speech in a variety of situations and has a certain degree of anti-compressibility, but it fails to show its effectiveness for resampled speech with a small tampering amplitude. Zhang et al. [19] proposed an algorithm that uses the local binary pattern (LBP) operator to model the statistical change characteristics of spectrograms and uses support vector machines (SVMs) for resampling detection, which has better detection performance and better robustness against MP3 compression, but the accuracy of detection for resampling factors, ranging from 0.8 to 1.2, still needs to be improved. Although the methods discussed above can achieve good accuracy, most require artificial feature engineering, which involves massive human professional knowledge and is usually a very time-consuming process. In recent years, deep learning has made significant progress in machine learning and has been widely applied in various fields such as digital image recognition, speech recognition, and steganography analysis [20,21,22,23,24]. Among them, Xu et al. [20] proposed a multiscale attention network for splicing tampering forensics in image recognition, which utilizes the integration of residual attention and multiscale information in order to improve the detection accuracy. Lang et al. [21] applied deep learning technology to the field of industrial defect detection, aiming to improve the accuracy of magnetic flux leakage (MFL) image recognition of pipeline corrosion defects, and achieved remarkable results. Taiba et al. [22] proposed a stride-based convolutional neural network (SCNN) model for speech emotion recognition. Banerjee et al. [23] applied deep learning in bio-signal steganography to provide robust, undetectable and trustworthy information security techniques. Huang et al. [24] built an attention-guided robust image watermarking model based on a generative adversarial network (GAN) to solve the problem of insufficient features and failure to highlight essential features in the existing deep learning watermarking models.

Traditional features have certain advantages in some aspects. But if the scene used is changed, the detection effect of manual features may be reduced [25,26,27]. In contrast, deep learning has great advantages at the level of extracting features, and it also has impressive classification capabilities. However, less work has been performed on applying deep learning to speech-tampering forensics. As can be seen in Refs. [28,29,30], some studies exist on speech copy–move detection, speech splicing detection, and speech deletion detection using deep learning. In these studies, the two-dimensional graph generated by speech signals is used as the input of a neural network and its graphical characteristics are utilized to achieve different speech tasks. The two-dimensional graph in Figure 1 is also known as the spectrogram, which is a simple but informative speech visualization map. In addition to being easier to interpret than the speech signal itself, it preserves time, frequency and amplitude information in a single plot. Therefore, using deep learning methods to extract time–frequency features of spectrograms for speech resampling detection will to some extent help us improve the detection accuracy of forged speech.

Based on the existing research, this paper conducts experiments using a resampling-tampered corpus with multiple factors produced by Zhang et al. [19] utilizing the TIMIT dataset. Combined with the deep learning method, this paper constructs a feature pyramid network (FPN) structure to extract the features of the spectrogram, carrying out feature fusion, and adds the Squeeze and Excitation (SE) attention mechanism [31] to obtain the feature map containing low-level location information and high-level semantic information for resampling-tamper detection. In summary, the contributions of this paper mainly include the following three points:

• We propose to use a deep learning method to extract resampling features from spectrograms containing time–frequency information, and apply various models of classical image classification for speech resampling detection;
• We apply the SE attention mechanism to the feature pyramid fusion network, which can focus on those channels that are more efficient for the resampling classification task and help the network to make resampling decisions;
• Thorough experimental evaluations show that the proposed method is suitable for speech detection tampered with multiple resampling factors, especially for tampering close to the original speech, and the detection performance is significantly improved. Moreover, the effect of MP3 compression on speech resampling detection is evaluated, and the results show that the proposed model is robust to MP3 compression.

The remaining sections of this paper are as follows: the second part focuses on speech preprocessing and introduces the structure of our proposed network model; the third part is about the experimental setup and analysis of the experimental results. Finally, in the fourth part, the paper is summarized and future work is prospected.

2. Proposed Method

The overall framework of the proposed approach is shown in Figure 2. It consists of three modules: speech signal extraction and preprocessing, feature extraction and fusion, and classification. The modules are described in detail in the following sections.

2.1. Speech Preprocessing

2.1.1. Audio Signal to Spectrogram

Since a one-dimensional speech signal cannot adequately represent the features of resampled speech, this paper converts it into a two-dimensional spectrogram to represent three-dimensional features [32,33]. The x-axis represents time, and the y-axis represents frequency. The color of the coordinate points in the spectrogram represents the level of volume. For a speech signal, its changes are rapid, and the Fourier transform is suitable for analyzing stable signals. It is necessary to slice the audio of variable length into fixed-length segments, namely, framing. In order to ensure a smooth transition between frames to maintain continuity, there will be overlap. Before the Fourier transform of each frame of speech, it needs to be multiplied by a smooth window function, so that both ends of the frame can be smoothly attenuated to zero. The purpose of this is to avoid spectrum leakage and generate a higher quality spectrum [34]. Subsequently, a fast Fourier transform (FFT) is performed on each frame of the speech signal, and its transformation form is as follows:

$$X(k,t)=\sum _{n=0}^{N-1}{w}_n{x}_t\left(n\right)e^{\frac{-j2\pi kn}{N}};k=0,1,2,\dots ,N-1$$

(1)

where $x_t\left(n\right)$$(t=1,2,...)$ is the signal of the t-th frame after the speech signal is divided into frames, $w_n$ is the selected Hamming window function, and N is the number of frames. The transformation of each obtained frame is spliced according to the frequency axis to generate a linear spectrum. Finally, in order to reduce the dynamic change of the amplitude component, the linear spectrum is converted into a logarithmic spectrum:

$$S(k,t)=log\left|X\right(k,t\left)\right|$$

(2)

The final spectrogram is calculated by the above equation. In this paper, the frame length is 50 ms and the frame shift is 25 ms in the process of converting speech signals from the time domain to the frequency domain.

2.1.2. The Performance of Resampling on the Spectrogram

Resampling can be achieved by up-sampling, interpolation, and down-sampling, depending on the desired sampling multiple. For any multiple of sampling, a combination of up-sampling, interpolation, and down-sampling is required. Assuming an audio signal $x\left(n\right)$ with a length of m sampling points, p is the interpolation factor, and q is the decimation factor. After resampling by the sampling factor $p/q$, the number of samples of the signal can be increased or decreased. The process is as follows:

• Up-sampling: By inserting $p-1$ zeros between every two sample points of the signal, the sampling point of the signal $x_u\left(n\right)$ becomes $p(m-1)+1$ after the change, so that the signal $x\left(n\right)$ is expanded by a factor of p in the time domain. When $n=1,2,...,m$, $x_u(p(n-1)+1)=x\left(n\right)$; otherwise, $x_u\left(n\right)=0$.
• Interpolation: The signal $x_u\left(n\right)$ is convolved by a low-pass linear interpolation filter $h\left(n\right)$ to obtain the interpolated signal $x_c\left(n\right)=x\left(n\right)\ast h\left(n\right)$. According to the use of different filters [35], the interpolated signal can be divided into linear interpolation, spline interpolation, and cubic interpolation.
• Down-sampling: One point is extracted every $q-1$ sample points, so that the signal $x_c\left(n\right)$ is reduced by a factor of q in the time domain. The changed signal is $x_d\left(n\right)=x_c(1+q(n-1))$. When $n=1,2,...,\frac{(p-1)m}{p}$,$\frac{(p-1)m}{p+1}$, the resampled signal is $y\left(n\right)=x_d\left(n\right)$.

Figure 3 shows the spectrogram of the original speech signal that has undergone resampling by a factor of 2. Different resampling results are obtained based on different interpolation filters $h\left(n\right)$. It is observed that the length of the original signal is doubled in the time domain, the low-frequency region has high resolution and the bandwidth is narrowed, which is affected by a certain degree of noise and distortion. After applying the linear interpolation filter, there are jagged changes in the spectrogram, and these are prominent in the high-frequency region. Spline interpolation is an interpolation method based on local polynomial approximation, which is more accurate and complex than linear interpolation. From the analysis of the spectrogram, it alleviates the jagged changes in the high-frequency region to a certain extent. Cubic interpolation based on cubic polynomial approximation is a more accurate interpolation method than the previous two. It can be seen that after cubic interpolation, a smooth spectral curve is obtained, which can better analyze and process speech signals.

2.2. Image Preprocessing

After the speech preprocessing operation, the speech spectrogram I is obtained, where $I_{:,\theta }=(I_{1,\theta },\dots ,I_{F,\theta })$ is the $\theta$-th frame, $\theta \in (0,T)$, F represents the number of frequency bins extracted, and T represents the number of time frames. The spectrogram size is $F\times T$, and is saved in the PNG format. After the speech with 3 s duration is processed with a frame length of 50 ms and a frame shift of 25 ms, the time frame $T_1=188$ and the frequency bin $F_1=513$. Since the average duration of the TIMIT speech database is 3 s, in order to preserve the information of the resampled spectrogram in the frequency direction, we random-crop the spectrogram I of a different size to the size of $F_1\times T_1$ along the time direction, which is used as the input to the network. When $T<T_1$, the pixels will be interpolated with zero-padding, that is, $pixel\left(I_{1:F_1,:t},I_{1:F_1,t+T:T_1}\right)=0$, where the parameter t is a random number selected, $t\in (0,T_1-T)$. In contrast to resizing, which can distort the temporal and spectral characteristics of the speech signal, random-cropping preserves the original resolution and structure of the spectrogram, ensuring that important features are not lost or altered. Additionally, random-cropping helps to prevent overfitting during model training, so that the model can be better generalized to new data [36]. Before entering the network training and testing, the spectrogram needs to be standardized to prevent the gradient from disappearing or exploding, and to speed up the training and fitting process.

2.3. Architecture

The proposed model contains a feature extraction module, a feature fusion module and a classification module. The feature extractor is a bottom-up module in the feature pyramid that is stacked using the Bottleneck infrastructure of ResNet. The feature fusion module fuses the features of four different scales, and the obtained feature map contains information of different levels. Then, through the SE attention mechanism, greater weight is given to those channels that are more effective for classification. After the feature map is up-sampled, the lateral connection operation in the feature pyramid is performed to achieve cross-scale information transfer, so as to improve the performance of resampling detection tasks [37]. The classification module uses multiple full-connection layers to achieve feature dimension reduction, and obtains more abstract and distinguishable feature vectors, which are used for resampling decisions.

2.3.1. Feature Extraction

(a)

Bottleneck: A residual basis of ResNet [38], which is different from another residual basis, Basicblock, Bottleneck is divided into three steps: first, $1\times 1$ convolution is used to reduce the dimension of data; then the conventional $3\times 3$ convolution kernel is used; and finally, $1\times 1$ convolution is used to increase the dimension of the data. Its structure is similar to an hourglass, so it is called a bottleneck structure. For example, if the number of input channels is 256, the $1\times 1$ convolutional layer will first reduce the number of channels to 64, and then, increase the number of channels to 256 after the $3\times 3$ convolutional layer.

(b)

FPN: The feature pyramid is a pyramid shape composed of features at different scales extracted by deep convolutional neural networks [39]. In this paper, we use four bottleneck modules as the backbone network for extracting features at different scales, and the high-level features with low resolution but high-semantic information and the low-level features with high resolution but low-semantic information are added by up-sampling and lateral connection methods. The features at all scales have rich semantic information. Finally, feature maps of different scales are used for classification tasks.

2.3.2. Feature Fusion

(a)

Concat: Unifies the scale of the feature map through the maximum pooling operation. Utilizes the $1\times 1$ convolution kernel to adjust the 4-layer features to the same number of channels, and feature fusion is completed by using the Concat function.

(b)

SENet: Adjusts the channel attention of the fused feature map through the SE attention module to enhance the expression ability of the feature map [40]. SENet, shown in Figure 4, is a plug-and-play channel attention module that is often applied in visual models. It can enhance the channel features of the input feature map, and the output of the SE module does not change the size of the input feature map. SENet mainly includes two operations: squeeze and excitation.

• $S\left(F_{sq}(·)\right)$: Squeeze refers to compressing the spatial information in the input feature map. The input feature map $\mathit{U}$’s dimensions are $H\times W\times C$. In the spatial dimension, global average pooling is realized to obtain a $1\times 1\times C$ feature map. The following equation is used:

  $$z_c=F_{sq}\left(u_c\right)=\frac{1}{H\times W}\sum _{i=1}^H\sum _{j=1}^W{u}_c(i,j)$$

  (3)
• $E\left(F_{ex}\right)$: Excitation combines the learned channel attention information with the input feature map $\mathit{Z}$. The vector obtained in the previous step is processed through two fully connected layers $W_1$ and $W_2$, and the channel weight value $\mathit{s}$ is obtained. The formula is as follows:

  $$s=F_{ex}(z,W)=\sigma \left(g(z,W)\right)=\sigma \left(W_2\delta \left(W_{1}z\right)\right)$$

  (4)
  The activation function of the first fully connected layer is ReLU, and the activation function of the second fully connected layer is sigmoid.
• $Scale\left(F_{scale}\right)$: The generated weight vector $\mathit{s}$ is weight-assigned to the feature map $\mathit{U}$. To obtain the feature map $\tilde{\mathit{X}}$ with channel attention, the formula is as follows:

  $$\widetilde{{\mathbf{X}}_C}=F_{scale}(u_c,s_c)=s_c{u}_c$$

  (5)
  The size of $\tilde{\mathit{X}}$ is the same as the feature map $\mathit{U}$, which realizes the weighting of the feature map $\mathit{U}$. The SE attention module can enhance the resampling time–frequency characteristics expressed in the feature map by focusing on channels that are effective for the resampling classification task and suppressing channels that are ineffective.

(c)

Lateral connection: Used in the top-down up-sampling process of the network, accompanied by the operation of lateral connection. The results of up-sampling are fused with the bottom-up-generated feature maps of the same scale, that is, added pixel by pixel [40]. In the experiment, bilinear interpolation is selected for up-sampling. $\tilde{\mathit{X}}$ is obtained by the SE attention module, which reduces the number of channels to 256 by $1\times 1$ convolution, obtaining ${\tilde{\mathit{X}}}^{\prime }$ through (6):

$${\tilde{\mathit{X}}}^{\prime }(i^{\prime },j^{\prime })=\sum _{i=0}^1\sum _{j=0}^1{\omega }_{ij}\tilde{\mathit{X}}(i+i^{\prime },j+j^{\prime })$$

(6)

where ${\omega }_{ij}$ represents the weight coefficients of the four pixels $\tilde{X}(i^{\prime },j^{\prime }),\tilde{X}(i^{\prime }+1,j^{\prime }),$$\tilde{X}(i^{\prime },j^{\prime }+1),\tilde{X}(i^{\prime }+1,j^{\prime }+1)$ closest to the target pixel, which can be obtained from the distance between the target pixel and these four pixels. We will process the feature maps extracted from the Bottleneck3 through $1\times 1$ convolution, reducing the number of channels to 256, and obtain $I^{\prime }$. After Equation (7), the final feature map $\mathit{F}$ is calculated.

$$F={\tilde{X}}^{\prime }+I^{\prime }$$

(7)

The main purpose of this is to exploit detailed information such as edges and textures of shallow resampled speech, which fuses semantic information of deep feature maps to achieve cross-level information transfer.

2.3.3. Classification and Evaluation Metrics

In the experiments, the obtained feature map $\mathit{F}$ is flattened to remove the spatial relationship between pixels and effectively combine the spatial information with the channel information. After the fully connected layer, the high-dimensional feature vectors are compressed to be low dimensional to obtain more abstract and distinguished feature vectors and to reduce the computational complexity and the number of parameters. Using the ReLU activation function, the nonlinear fitting ability of the model is increased, and then, the probabilities of two classes are output after the linear layer. Finally, the proposed model loss is calculated through the cross-entropy loss function.

Cross-entropy: Cross-entropy is mainly used to measure the difference between two probability distributions, which is a loss function often used in classification problems. Firstly, log-softmax normalization is performed on the obtained vector, and the formula is as follows:

$$log{p}_i=logSoftmax\left(z_i\right)=log\frac{e^{z_i}}{{\sum }_{c=1}^C{e}^{z_c}}$$

(8)

Given the output node i, $p_i$ is the probability of the correct category. Next, the NLLloss is calculated as in Equation (9):

$$nllloss=-\frac{1}{N}\sum _{i=1}^N{y}_i\left(log{p}_i\right)$$

(9)

$y_i$ is the data label after one-hot encoding, 1 for the positive class and 0 for the negative class. Finally, combining the two gives the cross-entropy loss:

$$C=-\frac{1}{N}\sum _{i=1}^N{y}_i\left(log{p}_i\right)+(1-y_i)log(1-p_i)$$

(10)

The predicted probability for each category is $p_i$ and $1-p_i$. C is one of the evaluation metrics used in this paper.

2.4. Changes in Spectrogram Features in Different Network Layers

To visually observe the ability of the proposed model to extract features and capture resampling changes, we analyzed the feature variations in the spectrogram at different network layers. Figure 5 shows the feature visualization process of the spectrogram through the proposed model, which includes down-sampled speech, original speech, and up-sampled speech. To observe the changes more intuitively, the resampling is performed by the zero-complementing interpolation method. When p/q = 0.6, the time-domain length decreases, the frequency-band length stretches, and the energy of the high-frequency component in the spectrogram increases. And zero-padding is carried out before inputting the model for training. When p/q = 2, the length doubles in the time domain. As the frequency resolution increases in the low-frequency region, the frequency bandwidth decreases. Therefore, the frequency domain exhibits the characteristics of compression, resulting in the energy reduction of the high-frequency components in the spectrogram.

When the spectrogram enters the first convolution layer, the proposed model can capture the resampling characteristics of the above analysis. After the maximum pooling layer, the model focuses on the pixels with large energy and reduces the dimensions of the feature map. The output of Bottleneck2 layer indicates that the model is still able to capture the characteristics of up-sampling, original speech, and down-sampling in the time and frequency domains. The model extracts semantic information of the spectrogram through Bottleneck4, and the features of the down-sampling frequency domain stretching and up-sampling frequency domain compression can still be observed. It is noted in the figure that the SE layer helps to assign greater weight to channels that are more effective for model detection tasks, and the pixel values of the feature map corresponding to channels are enlarged. Finally, after the lateral connection operation, due to the fusion of the edge texture information captured by the upper layer, it can be seen that the final feature map restores the characteristics of the resampled spectrogram in the time and frequency domains and contains rich semantic information. Thus, the subsequent resampling classification task has better performance.

3. Experiments and Analysis

3.1. Experimental Setup

3.1.1. Dataset

Since there is no public speech database for speech resampling detection at present, the resampling corpus constructed by Zhang et al. [19] is used in this paper, and the original speech is selected from the TIMIT dataset. The TIMIT database contains 6300 utterances, with an average duration of 3 s, from 630 participants in eight major dialect areas in the United States. The original speech files is mono, PCM encoded, 16 kHz sampling ratio, and 16 bit quantization [18]. The resampling corpus selects 3000 original speech samples from TIMIT. These samples were resampled using the linear interpolation, spline interpolation, and cubic interpolation methods, and the sampling factor range was 0.6∼1.6 with a step size of 0.1. This resampled corpus is used to comprehensively evaluate the effectiveness of our proposed method for detecting resampled speech.

As shown in

Table 1. Corpus used in experiments.

Dataset	Original Speech	Resampled Speech (0.6∼1.6)
	Un-Interpolated	Linear	Spline	Cubic	Mixed
Uncompressed	3000 samples	3000 × 3 × 10 = 90,000 samples			3000 × 10 = 30,000 samples
Compressed (32 kbps, 64 kbps, 128 kbps)	3000 × 3 = 9000 samples	3000 × 3 × 10 × 3 = 270,000 samples			3000 × 3 × 10 = 90,000 samples

, the corpus also includes MP3-compressed speech databases using three different bit rates of 32 kbps, 64 kbps, and 128 kbps for the original speech and resampled speech, as well as the mixed corpus constructed by randomly selecting resampled speech from different interpolation databases.

3.1.2. Implementation Details

The experiment used the Pytorch framework, the development environment was python 3.7, and the hardware platform was an NVIDIA GTX 3070. The study in this paper was experimented on a corpus made by Zhang [19] et al. based on the TIMIT database. The detection experiment of each sampling factor has a total of 6000 pictures. The training set and the test set are divided into 5:5, and the pictures of variable length are randomly cropped, and then, standardized. As supervised learning, instances of the false speech class are marked 1 as negative and instances of the natural speech class are marked 0 as positive. Model training uses cross-entropy as a loss function and using SGD optimizer. The initial learning rate is 0.01, the batch size is 16, and the number of training epochs is set to 12. In the experiment, we focus on selecting the mixed corpus for detection to verify the effectiveness of the model. In order to verify the robustness of the model, we selected speeches with different compression bit rates in the mixed corpus for comparative experiments.

3.2. Results and Analysis

3.2.1. Different Interpolation Methods

The process of resampling was introduced earlier, including up-sampling, interpolation, and down-sampling. The interpolation methods mainly include linear interpolation, spline interpolation, and cubic interpolation. Different interpolation methods have different effects on the accuracy of resampling-tamper detection. When the resampling factors are 0.8, 0.9, 1.1, and 1.2, the tampered speech is closer to the original speech, which has higher reconstruction quality and less information loss. Therefore, the tamper detection accuracy of these factors is low in the current methods. In this section, the proposed model is compared with a traditional speech signal feature extraction algorithm [16] and a manual spectrogram feature extraction algorithm [18,19] on the corpus of three interpolation methods and four representative resampling factors. The restuls are shown in

Table 2. Resampling detection results under different interpolation methods.

$\mathrm{p}/\mathrm{q}$	$\mathbf{Method}$	$\mathbf{Mixed}\mathbf{Interpolation}$	$\mathbf{Linear}\mathbf{Interpolation}$	$\mathbf{Spline}\mathbf{Interpolation}$	$\mathbf{Cubic}\mathbf{Interpolation}$
0.8	Hou et al. [16]	0.7061	0.7237	0.7061	0.6841
	Wang et al. [18]	0.4943	0.4820	0.5310	0.4950
	Zhang et al. [19]	0.8553	0.8360	0.8340	0.8270
	$\mathbf{Proposed}\mathbf{Method}$	$\mathbf{0.9657}$	$\mathbf{0.9573}$	$\mathbf{0.9610}$	$\mathbf{0.9656}$
0.9	Hou et al. [16]	0.6876	0.6771	0.6619	0.6673
	Wang et al. [18]	0.4490	0.4140	0.4840	0.4550
	Zhang et al. [19]	0.6960	0.7030	0.6600	0.6950
	$\mathbf{Proposed}\mathbf{Method}$	$\mathbf{0.8908}$	$\mathbf{0.8982}$	$\mathbf{0.8883}$	$\mathbf{0.8900}$
1.1	Hou et al. [16]	0.6902	0.6514	0.6806	0.6817
	Wang et al. [18]	0.6247	0.6300	0.6690	0.6700
	Zhang et al. [19]	0.7413	0.7530	0.7440	0.7430
	$\mathbf{Proposed}\mathbf{Method}$	$\mathbf{0.9767}$	$\mathbf{0.9810}$	$\mathbf{0.9910}$	$\mathbf{0.9720}$
1.2	Hou et al. [16]	0.6657	0.6745	0.6201	0.6675
	Wang et al. [18]	0.7020	0.6620	0.7490	0.7030
	Zhang et al. [19]	0.8127	0.8160	0.8460	0.8000
	$\mathbf{Proposed}\mathbf{Method}$	$\mathbf{0.9930}$	$\mathbf{0.9770}$	$\mathbf{0.9986}$	$\mathbf{0.9834}$

.

• Different interpolation methods have a certain impact on the detection model, and the detection accuracy obtained by different interpolation methods varies from 0.17% to 2.16%. In general, the detection accuracy of the model is less affected by the interpolation method.
• Regardless of the detection under different sampling factors or different interpolation methods, the accuracy of the method proposed in this paper is higher than the methods of Hou [16], Wang [18] and Zhang [19]. The results show that the performance of using the deep learning method to extract spectrogram features for resampling detection is significantly better than the existing research results.

3.2.2. Architecture Comparison

With the innovation of deep learning, the ability of neural networks to extract features is also constantly enriched. In this paper, one-dimensional speech signals are converted into a two- dimensional spectrogram, and neural networks are used to extract the spectrogram features, which greatly improves the ability of speech resampling detection. In this section, after attempting to train different network models, we construct a deep attention network that combines low-level location features with high-level semantic features and adds SE attention modules. In the experiment, classic classification networks are compared with the proposed model to test the accuracy of resampling detection. ResNet14 is a feature extractor formed by stacking four Bottlenecks, and FPN is a model structure that integrates top-down two-layer feature maps based on ResNet14. The speech of the four sampling factors in the table is from the mixed corpus.

The data in

Table 3. Detection results of different network models.

$\mathbf{p}/\mathbf{q}$	$\mathbf{AlexNet}$	$\mathbf{ResNet}14$	$\mathbf{FPN}$	$\mathbf{The}\mathbf{Proposed}\mathbf{Model}$
0.8	0.9495	0.9470	0.9595	$\mathbf{0.9657}$
0.9	0.5758	0.7650	0.8614	$\mathbf{0.8908}$
1.1	0.7504	0.9003	0.9664	$\mathbf{0.9767}$
1.2	0.9742	0.9714	0.9909	$\mathbf{0.9930}$

and Figure 6 show the impact of the four network models on the accuracy of different resampling factors. It is worth noting that when p/q = 0.8 and p/q = 1.2, AlexNet, ResNet14, and FPN all have good detection capabilities. When the resampling factor is 1.1, it is observed that the classification ability of the AlexNet model decreases, and the detection performance of ResNet14 also slightly weakens. When p/q = 0.9, the detection accuracy of the four models has a different degree of decline, among which AlexNet and ResNet14 decrease significantly, indicating that the two comparison models could not adapt to tamper detection of multiple resampling factors. The accuracy rate of the proposed model is as high as 89.09%, which is about three percentage points higher than the high-performance FPN. In addition, under other sampling factors, the proposed model achieved optimal results, which verifies its effectiveness.

Figure 7 shows the training loss curves of different network models when the resampling factor is 1.1. Its variation indicates that the loss value of the proposed model can converge quickly and maintain stability with the increase in the number of iterations.

To validate the stability of the proposed model, we evaluated the performance using different backbone networks in the speech resampling detection model. The choice of backbone network is crucial for the model’s feature extraction efficiency and overall performance. In this study, we used ResNet14 as the backbone for the proposed model. Additionally, we integrated other backbone networks, EfficientNet and Vision Transformer (ViT), into our model framework and analyzed their performance in detection accuracy. Table 1 shows the impact of different backbone networks on the detection accuracy under small-scale resampling-tampering.

As can be seen from

Table 4. Comparison of detection results for different backbone networks.

$\mathbf{p}/\mathbf{q}$	$\mathbf{EfficientNet}\text{-}\mathbf{B}0$	$\mathbf{ViT}$	$\mathbf{ResNet}14$
0.8	0.9270	0.9533	$\mathbf{0.9657}$
0.9	0.8510	0.8867	$\mathbf{0.8908}$
1.1	0.9240	0.9397	$\mathbf{0.9767}$
1.2	0.9620	0.9826	$\mathbf{0.9930}$

, the ResNet14 backbone has better detection performance than the other two backbone networks. Additionally, we experimented with different learning rates and found that ResNet14 could adapt to various learning rates. This adaptability effectively helped us complete the speech resampling detection task.

It is worth noting that the results for ViT were achieved using a learning rate ten times lower than those used for ResNet14 and EfficientNet-B0, as ViT could not be trained properly with the same learning rate. Therefore, although ViT showed high accuracy when p/q = 0.9, its requirement for a lower learning rate indicates potential training instability issues under different learning rates.

3.2.3. Different Compression Sampling Factors

In this section, our main objective is to compare the detection capabilities of the proposed model with traditional methods under different sampling factors. In traditional methods, Hou et al. [16] calculated the second-order difference of the speech signal and found the periodicity of the zero-value interval to detect resampling. Wang et al. [18] found that an incomplete interpolation process resulted in inconsistent bandwidth and sampling frequency of the spectrogram to detect resampling. Zhang et al. [19] used LBP operators to model time–frequency statistical changes in the spectrogram, and used SVM to classify the features of the LBP histogram to distinguish between original speech and resampled speech. In the case of large-scale speech tampering, the above methods are able to determine speech resampling to some extent. However, for small-scale tamper detection, that is, the resampling factor is 0.8 to 1.2, they do not show high detection accuracy. Therefore, the experiment in this section compares the detection accuracy of 10 resampling factors using the proposed model and other methods on the mixed resampling corpus, as shown in the following table.

The results in

Table 5. Detection results of different network models.

$\mathbf{p}/\mathbf{q}$	$\mathbf{Hou}\mathbf{et}\mathbf{al}.$ [16]	$\mathbf{Wang}\mathbf{et}\mathbf{al}.$ [18]	$\mathbf{Zhang}\mathbf{et}\mathbf{al}.$ [19]	$\mathbf{The}\mathbf{Proposed}\mathbf{Model}$
0.6	0.7120	0.6593	0.9813	$\mathbf{0.9988}$
0.7	0.6884	0.5573	0.9407	$\mathbf{0.9916}$
0.8	0.7061	0.4943	0.8553	$\mathbf{0.9657}$
0.9	0.6876	0.4490	0.6960	$\mathbf{0.8908}$
1.1	0.6902	0.6247	0.7413	$\mathbf{0.9767}$
1.2	0.6657	0.7027	0.8127	$\mathbf{0.9930}$
1.3	0.6603	0.7090	0.9093	$\mathbf{0.9961}$
1.4	0.6523	0.7617	0.9433	$\mathbf{0.9971}$
1.5	0.6346	0.7873	0.9649	$\mathbf{0.9949}$
1.6	0.6804	0.8373	0.9791	$\mathbf{0.9986}$

show that the detection accuracy of our proposed model is better than other methods under a resampling factor of 0.6∼1.6. Moreover, the accuracy under resampling factors of 0.8, 0.9, 1.1, and 1.2, which are difficult to detect, is greatly improved. In particular, the accuracy of resampling-tampering detection with p/q = 0.9 is increased by nearly 20%. And with an increase or decrease in the resampling factor, the accuracy rate gradually improves. The reason is that the larger the scale of down-sampling and up-sampling, the more obvious are the time–frequency variation characteristics displayed on the spectrogram. The model can effectively learn its changes and make better classification decisions, which verifies that the proposed model has a strong ability to extract resampling features from the spectrogram, and has high generalization.

3.2.4. The Impact of MP3 Compression

Speech compression is necessary in the process of voice transmission. In real-world scenarios, MP3 is the most commonly used speech format, which is obtained by compressing WAV speech [41]. However, the MP3 compression operation may obscure previous resampling traces in speech, so we test the robustness of MP3 compression in this section.

The experiment uses compression bit rates of 128 kbps, 64 kbps, and 32 kbps to compress speech signals, as shown in Figure 8. As the compression bit rate decreases, the frequency resolution of the high-frequency region in the spectrogram decreases, resulting in a more ambiguous representation of high-frequency signals and a relatively clearer representation of low-frequency signals [42]. A low compression bit rate leads to distortion of the speech signal, which makes abnormal spectral components appear in the spectrogram.

In order to analyze whether the changes in the spectrogram caused by MP3 compression has some influence on the resampling detection model in this paper, we train and test the original speech and resampled speech with different compression bit rates. The experiments analyze the changes in the model detection effect after the original speech and the resampled speech of each factor are compressed, as shown in

Table 6. Comparison of detection accuracy under different compression bit rates.

$\mathbf{Bit}\mathbf{Rate}$		$\mathbf{p}/\mathbf{q}$	$0.6$	$0.7$	$0.8$	$0.9$	$1.1$	$1.2$	$1.3$	$1.4$	$1.5$	$1.6$
	Method
32 kbps	Hou et al. [16]		0.6984	0.6977	0.6968	0.7222	0.7303	0.7485	0.6549	0.6596	0.6510	0.6393
	Wang et al. [18]		0.6593	0.5447	0.4977	0.4387	0.6683	0.7187	0.7433	0.7733	0.8110	0.8660
	Zhang et al. [19]		0.9980	0.9787	0.9627	0.9500	0.9540	0.9580	0.9633	0.9760	0.9813	0.9913
	$\mathbf{The}\mathbf{Proposed}\mathbf{Model}$		$\mathbf{0.9992}$	$\mathbf{0.9950}$	$\mathbf{0.9785}$	$\mathbf{0.8961}$	$\mathbf{0.9002}$	$\mathbf{0.9642}$	$\mathbf{0.9808}$	$\mathbf{0.9947}$	$\mathbf{0.9951}$	$\mathbf{0.9978}$
64 kbps	Hou et al. [16]		0.7059	0.7219	0.7061	0.6702	0.6800	0.6752	0.6482	0.6596	0.6407	0.6739
	Wang et al. [18]		0.6330	0.5297	0.4983	0.4677	0.6367	0.6833	0.7097	0.7587	0.7943	0.8400
	Zhang et al. [19]		0.9860	0.9407	0.8473	0.7040	0.7380	0.8273	0.9047	0.9480	0.9733	0.9880
	$\mathbf{The}\mathbf{Proposed}\mathbf{Model}$		$\mathbf{0.9991}$	$\mathbf{0.9910}$	$\mathbf{0.9574}$	$\mathbf{0.8676}$	$\mathbf{0.8857}$	$\mathbf{0.9504}$	$\mathbf{0.9801}$	$\mathbf{0.9934}$	$\mathbf{0.9974}$	$\mathbf{0.9983}$
128 kbps	Hou et al. [16]		0.7179	0.7346	0.6963	0.6795	0.6809	0.6884	0.6473	0.6464	0.6299	0.6543
	Wang et al. [18]		0.6370	0.5273	0.4993	0.4587	0.6397	0.6840	0.7097	0.7513	0.7973	0.8403
	Zhang et al. [19]		0.9860	0.9373	0.8440	0.6847	0.7140	0.8187	0.8980	0.9407	0.9727	0.9840
	$\mathbf{The}\mathbf{Proposed}\mathbf{Model}$		$\mathbf{0.9989}$	$\mathbf{0.9912}$	$\mathbf{0.9572}$	$\mathbf{0.8293}$	$\mathbf{0.8843}$	$\mathbf{0.9486}$	$\mathbf{0.9756}$	$\mathbf{0.9920}$	$\mathbf{0.9972}$	$\mathbf{0.9975}$

.

It can be observed from Table 6 and Figure 9 that the performance of the model is disturbed after MP3 compression, and the detection performance of the algorithm is degraded to some extent, but the proposed method still has a high detection accuracy for compressed resampled speech. By observing the feature map captured by the model, it is found that the smaller the speech compression bit rate, the more significant the distortion and noise effects of the signal, but the classification ability of the model is improved. This is because the sampling rate of the original speech does not change before compression, and the degree of information loss is relatively light, while the resampled speech being compressed results in more information loss due to the sampling rate conversion before compression, so that the difference between the two is more significant.

However, at a compression bit rate of 32 kbps, two types of resampled speech close to the original speech have a lower detection capability than the method of Zhang et al. [19] because the LBP operator is a local feature descriptor, which has a better representation of high-frequency detail loss, and the proposed model focuses on learning more complex global time–frequency variation features. Compared with other methods, for which the resampling detection performance degrades at high compression bit rates, the detection accuracy of the proposed model is less affected by the degree of compression, indicating that the proposed method is strongly resistant to compression.

4. Conclusions

The application of deep learning methods in speech-resampling-tamper detection is a new task. Based on our previous work [19], this paper uses a feature pyramid network that integrates attention mechanism to extract the spectrogram features of original and tampered speech and classify them. The experimental results show that the proposed method can distinguish between the original speech and the resampled speech effectively under the condition of speech tampering with multiple resampling factors. Among them, the accuracy of resampling detection is significantly improved when the amplitude of tampering is small. In addition, in the test under different compression ratios, the proposed model shows relatively good robustness, and can still accurately detect and recognize the compressed speech. To enhance the robustness and generalizability of our findings, we plan to extend our experiments to more recent and larger datasets such as DidiSpeech [43] and AiShell-4 [44]. We also plan to make our resampled and compressed speech corpus publicly available for further validation. For future work, we will consider using other deep networks to further improve the accuracy of small-scale resampling-tamper detection, and estimating the resampling factor is also a part of our future plans.