A Semi-Supervised Lie Detection Algorithm Based on Integrating Multiple Speech Emotional Features

Abstract

When people tell lies, they often exhibit tension and emotional fluctuations, reflecting a complex psychological state. However, the scarcity of labeled data in datasets and the complexity of deception information pose significant challenges in extracting effective lie features, which severely restrict the accuracy of lie detection systems. To address this, this paper proposes a semi-supervised lie detection algorithm based on integrating multiple speech emotional features. Firstly, Long Short-Term Memory (LSTM) and Auto Encoder (AE) network process log Mel spectrogram features and acoustic statistical features, respectively, to capture the contextual links between similar features. Secondly, the joint attention model is used to learn the complementary relationship among different features to obtain feature representations with richer details. Lastly, the model combines the unsupervised loss Local Maximum Mean Discrepancy (LMMD) and supervised loss Jefferys multi-loss optimization to enhance the classification performance. Experimental results show that the algorithm proposed in this paper achieves better performance.

1. Introduction

The rapid development of the Internet and social media has led to an information overload problem, including the spread of lies and misinformation. These phenomena can trigger significant negative consequences such as social unrest, political interference, and economic losses. Therefore, the development of lie detection technology is of great importance for building a more honest and secure social environment. Relevant research has already found wide application in critical fields such as law, the military, and forensics [1]. The psychological state of lying is a gradual and repetitive dynamic process. When people lie, they tend to become nervous, causing subtle changes in the vocal tract structure, with the fundamental frequency of the voice being higher than when telling the truth [2,3]. In the field of speech lie detection research, many acoustic features originally designed for speech emotion recognition have been confirmed to be equally crucial in identifying lies. These features can capture the tension, anxiety, or guilt that liars experience due to emotional fluctuations, providing key clues for lie detection by analyzing the pitch, rhythm, and intensity of speech. However, relying solely on these emotion-based acoustic features has limitations in revealing the complex psychological and physiological states of liars. To gain a deeper insight into deceptive behavior, it is essential to explore a broader set of features closely related to the act of lying, thereby enhancing the accuracy and depth of lie detection. There are some acoustic features in speech associated with lying [4]. These acoustic parameters change with varying stress levels, such as formants, Bark energy features, and Mel-frequency cepstral coefficients (MFCCs) [5,6]. Compared to the truth, lies exhibit fewer details, repetitive phrases, more content, shorter expression lengths, and inconsistencies in speech. Spectrum-based features can reflect the relationship between vocal tract shape and speech behavior [7]. Bareeda et al. [8] achieved an accuracy of up to 81% in lie detection using Mel-frequency cepstral features in “Real life trial data” dataset. Hanen et al. [9] extracted Mel-frequency cepstral features and pitch frequency, achieving good performance in lie detection. With the rise of deep learning, researchers have also started applying it to lie detection. Zhou et al. [10] proposed a deep belief network based on K-means and singular value decomposition (K-SVD) for extracting deep features, showing that deep features can better represent lies compared to traditional acoustic features. Sanaullah et al. [11] used Bark spectrum energy features and neural network technology to detect subtle differences between lies and the truth.

Although the aforementioned scholars have made significant achievements in the field of lie detection, data-driven deep neural networks heavily rely on large-scale labeled high-quality speech data. The lack of labeled data has become a critical issue constraining the development of the speech lie detection field. Semi-supervised learning algorithms provide a promising approach for speech lie detection tasks. Tarvainen et al. [12] proposed a mean-teacher model method, which optimizes semi-supervised learning by averaging model weights, simplifying the learning process of large datasets and improving test accuracy. Sun et al. [13] respectively trained the bidirectional long short-term memory networks (BILSTM) and support vector machine (SVM) model, and adopted the decision-level score fusion scheme to further integrate the classification results of each model. Fang et al. [14] proposed a multi-feature fusion speech lie detection algorithm based on attention mechanisms, constructing a hybrid model training network composed of a semi-supervised Denoising Auto Encoder (DAE) and a fully supervised Long Short-Term Memory(LSTM) network, effectively improving the accuracy of semi-supervised speech lie detection. Despite some achievements in semi-supervised research, the exploration of multi-feature lie detection algorithms under a semi-supervised framework has been neglected. A single feature cannot effectively represent lie information, so some scholars have considered combining multiple features for lie detection. Researchers at Columbia University considered combining lexical, prosodic, and acoustic features for lie detection research [15]. He et al. [16] proposed a Transformer-based deep hashing multi-scale feature fusion (TDH) algorithm. This algorithm utilizes hierarchical Transformers to capture both global and local features of images to obtain richer information. Praveen et al. [17] considered the complementarity between audio and video modalities and proposed an LSTM-based joint cross-attention model to capture intra-class and inter-class temporal dependencies. This paper aims to combine feature fusion methods with semi-supervised learning techniques to address the challenges in speech lie detection research, prevent dimensional disaster caused by multi-feature fusion when labeled data is insufficient, and fully utilize the potential feature information in unlabeled data to obtain robust feature representations in a semi-supervised learning environment.

Based on this, this paper proposes a semi-supervised lie detection algorithm based on integrating multiple speech emotional features. Firstly, LSTM network and self-coding network AE are used for feature extraction, followed by the introduction of joint attention method, which utilizes the correlation between different features to assist the feature fusion task and improve the accuracy of speech lie detection. Subsequently, Jefferys [18] loss is introduced to improve the supervised loss, which maximizes the target value of the output distribution and reduces the overfitting; Local Maximum Mean Discrepancy (LMMD) [19] is introduced to improve the unsupervised loss, which enables the model to learn richer feature information from unlabeled data. Experimental results demonstrate the effectiveness of the proposed algorithm.

2. Method

In this paper, we propose a semi-supervised lie detection algorithm based on integrating multiple speech emotional features, and the overall framework of the model is shown in Figure 1. First, the log Mel spectrogram features and acoustic statistical features are processed using LSTM network and self-coding network Auto Encoder (AE), respectively, to capture the contextual links between similar features, respectively. Second, the complementary relationship between different features is learned using the joint attention model to obtain feature representations with richer details. Subsequently, the Local Maximum Mean Difference (LMMD) is added to the unsupervised loss part to fully explore the unlabeled data’s potential feature structure information. Finally, the model is combined with supervised loss Jefferys for multi-loss optimization to improve the model’s classification performance.

2.1. Feature Processing

As described in [4,5,13,14], acoustic statistical features can effectively represent false information in speech, while deep learning algorithms can extract time-frequency domain features from log-Mel spectrogram [20]. So 312-dimensional acoustic statistical features $X_a$ based on artificial a priori knowledge and log-Mel spectrogram features $X_s$ commonly used in the field of speech lie detection are used, aiming at mining the correlation information within similar features and among different features, and exploring the effect of fusion of static statistical features and dynamic time-frequency features in the time domain. In this paper, acoustic statistical features contains Pitches, spectral_centroid, spectral_flatness, Mel Frequency Ceptral Coefficients (MFCC), chroma_stft, melspectrogram, spectral_contrast, zero_crossing_rate, Magnitudes, Root-mean-square energy (Rmse).

Let $X_a=\left\{x_a^1, x_a^2, \dots , x_a^O\right\}\in R^{d_a\times O}$ and $X_s=\left\{x_s^1, x_s^2, \dots , x_s^O\right\}\in R^{d_s\times O}$, where $d_a$ and $d_s$ denote the dimensions of acoustic statistical features and Mel spectrogram features, respectively, $x_a^i$ and $x_s^i$ is the single feature representation of all samples, and $i=1, 2, \dots , O$, o is the feature order number, O denoting the total number of feature samples.

Semi-supervised speech lie detection research requires a database containing a small number of labeled samples $D_l=\left\{X_l,Y_l\right\}$ and a database containing a large number of unlabeled samples $D_u=\left\{X_u\right\}$, where $X_l=\left(X_{al}, X_{sl}\right)$, $X_{al}=\left\{x_{al}^1, x_{al}^2, \dots , x_{al}^M\right\}\in R^{d_a\times M}$, $X_{sl}=\left\{x_{sl}^1, x_{sl}^2, \dots , x_{sl}^M\right\}\in R^{d_s\times M}$, $x_{al}^m$ and $x_{sl}^m$ are representations of labeled samples. $Y_l=\left\{y_l^1, y_l^2, \dots , y_l^m\right\}$ denote labels, $m=1, 2, \dots ,M$, M are the total number of labeled samples; $X_u=\left(X_{au}, X_{su}\right)$, $X_{au}=\left\{x_{au}^1, x_{au}^2, \dots , x_{au}^M\right\}\in R^{d_a\times N}$, $X_{su}=\left\{x_{su}^1, x_{su}^2, \dots , x_{su}^M\right\}\in R^{d_s\times N}$, $x_{au}^n$ and $x_{su}^n$ are representations of unlabeled samples, $n=1, 2, \dots , N$, N denote the total number of samples of unlabeled data, there $O=M+N$. In order to improve the classification accuracy and robustness of the semi-supervised model, in this paper, random noise enhancement technique is used to strongly enhance and weakly enhance the unlabeled data with the following formula:

$$x_{auweak}^n=x_{au}^n+{\lambda }_{auweak}·noise\left(x_{au}^n\right)$$

(1)

$$x_{austrong}^n=x_{au}^n+{\lambda }_{austrong}·noise\left(x_{au}^n\right)$$

(2)

$$x_{suweak}^n=x_{su}^n+{\lambda }_{suweak}·noise\left(x_{su}^n\right)$$

(3)

$$x_{sustrong}^n=x_{su}^n+{\lambda }_{sustrong}·noise\left(x_{su}^n\right)$$

(4)

where $x_{auweak}^n$ and $x_{austrong}^n$ denote the 312-dimensional acoustic statistical features that have been weakly enhanced by noise addition and strongly enhanced by noise addition, respectively, while ${\lambda }_{auweak}$ and ${\lambda }_{austrong}$ denote their weak enhancement coefficients and strong enhancement coefficients, respectively; $x_{suweak}^n$ and $x_{sustrong}^n$ denote the log Mel spectrogram features that have been weakly enhanced by noise addition and strongly enhanced by noise addition, respectively, while ${\lambda }_{suweak}$ and ${\lambda }_{sustrong}$ denote their weak enhancement coefficients and strong enhancement coefficients, respectively; and $noise\left(\right)$ is the Gaussian noise generating function, which generates the corresponding noise according to the shape of the original data noise data according to the shape of the original data.

2.2. Feature Fusion

In order to effectively utilize the dynamic time-frequency information in the speech signal, this paper adopts LSTM to process the log Mel spectrogram features $X_s$ to obtain more advanced depth frame-level feature $X_{sd}$. and processes the acoustic statistical features $X_a$ through the self-coding network AE to extract the deep statistical features $X_{ad}$ with stronger representations. Acoustic statistical features and Mel spectrogram features have a good complementary relationship, so this paper combines acoustic statistical features with Mel spectrogram features to capture finer-grained features.

First, the depth statistical features $X_{ad}$ and depth frame-level features $X_{sd}$ are simply spliced and fused to form the A-S joint feature vector $J=[X_{ad};X_{sd}]\in R^{d_j\times L}$, where $d_j=d_{ad}+d_{sd}$, denotes the dimension of the A-S joint feature vector J. Next, in order to obtain the semantic correlation between features and fully utilize the connection between similar features and different features, the combination of deep statistical features $X_{ad}$, deep frame-level features $X_{sd}$ and A-S joint feature vector J is considered to compute the joint correlation matrix. The joint correlation matrix of depth statistical features $X_{ad}$ and A-S joint feature vector J is shown in Equation (5):

$$C_a=tanh({\displaystyle \frac{X_{ad}^T{W}_{jad}J}{\sqrt{d_j}}})$$

(5)

where $W_{jad}$ denotes the learnable weight matrix mapping the relationship between the depth frame-level features $X_{ad}$ and the joint A-S feature vector J. Similarly, the joint correlation matrix of the depth frame level features $X_{ad}$ and the joint feature vector J can be obtained from Equation (6):

$$C_s=tanh({\displaystyle \frac{X_{sd}^T{W}_{jsd}J}{\sqrt{d_j}}})$$

(6)

where $W_{jsd}$ denotes the learnable weight matrix mapping the relationship between the depth frame-level features $X_{sd}$ and the joint A-S feature vector J.

Then, the attention weights of acoustic statistical features $X_a$ and log Mel spectrogram features $X_s$ are further calculated to help the feature fusion model focus on the important information and improve the efficiency of the model when learning the distribution of similar features. The attention weights of acoustic statistical features $X_a$ can be obtained by combining the joint correlation matrix $C_a$ and the deep statistical features $X_{ad}$, and the detailed calculation is shown in Equation (7).

$$H_a=\mathrm{ReLU}({\mathrm{W}}_{\mathrm{ad}}{\mathrm{X}}_{\mathrm{ad}}+{\mathrm{W}}_{\mathrm{ca}}{\mathrm{C}}_{\mathrm{a}})$$

(7)

where $H_a$ denotes the attention weights of the acoustic statistical features $X_a$, $W_{a}d$ and $W_{ca}$ is the learnable weight matrix.

The attention weights of the log Mel spectrogram features $X_s$ can be obtained by combining the joint correlation matrix $C_s$ and the depth frame-level features $X_{sd}$, and the detailed calculation is shown in Equation (6).

$$H_s=\mathrm{ReLU}({\mathrm{W}}_{\mathrm{sd}}{\mathrm{X}}_{\mathrm{sd}}+{\mathrm{W}}_{\mathrm{cs}}{\mathrm{C}}_{\mathrm{s}})$$

(8)

where $H_s$ denotes the attention weights of the acoustic statistical features $X_s$, $W_{s}d$ and $W_{cs}$ is the learnable weight matrix.

The attentional correlation features of the two types of features are computed by means of the attentional weights of the acoustic statistical features $X_a$ and the log-Mel spectrogram features $X_s$, with the formulas shown in (9) and (10) as follows.

$$X_{att,a}=H_a{W}_{ha}+X_{ad}$$

(9)

$$X_{att,s}=H_s{W}_{hs}+X_{sd}$$

(10)

where $W_{ha}$ and $W_{hs}$ denote the learnable weight matrices for acoustic statistical features $X_a$ and log Mel spectrogram features $X_s$, respectively.

Finally, the two types of attention-related features are combined to obtain the joint attention feature $X_{att}=\left[X_{att,a}{X}_{att,s}\right]$.

In order to obtain a better feature representation and reduce the influence of overfitting and other factors, time domain summation is used to ensure the information integrity and stability of the fused features; batch normalization, Gelu activator and Dropout are introduced to further optimize the joint attention features $X_{att}$. In the last part of the model, a softmax classifier is used to predict the classification results. Figure 2 shows the details of joint attention fusion.

2.3. Multi-Loss Optimization

Semi-supervised algorithms co-optimize the performance of semi-supervised models through supervised and unsupervised losses to obtain better classification performance.

2.3.1. Supervised Loss

Supervised loss provides the correct guidance direction for model training based on the supervised signals in the labeled data, which helps the model learn accurate prediction classification. In this paper, the Jefferys-based loss function is used, and the Jefferys scatter is added to the cross-entropy loss function to maximize the target value of the output distribution, while smoothing the non-target values to reduce overfitting. The formula is as follows:

$$L_s=-log\left(p_k\right)-\alpha {\displaystyle \frac{{\sum }_{i\ne k}log\left(p_i\right)}{K-1}}+\beta {\displaystyle \frac{{\sum }_{i\ne k}{p}_{i}log\left(p_i\right)}{1-p_k}}$$

(11)

wherh $p_k$ denotes the predicted probability of labeled samples after model training, K denotes the total number of categories, the non-targeted output values $p_i$ follow a uniform distribution, $\alpha$ and $\beta$ are hyperparameters.

2.3.2. Unsupervised Loss

Unsupervised loss introduces LMMD loss on the basis of cross-entropy loss function to constrain the distribution change of unsupervised features after different data enhancement to exclude the interference of redundant or irrelevant features. The formula is as follows:

$$L_{ce}={\displaystyle \frac{1}{M}}\sum _{m=1}^{M}1\left(max\left(p_m\right)\right)>mask\left(argmax\left(p_m\right)\right)·ce({\widehat{p}}_m,p_{pesudo})$$

(12)

where ${\widehat{p}}_m$ denotes the predicted probability of the strongly enhanced unlabeled feature after model training, $p_m$ denotes the predicted probability of the weakly enhanced unlabeled feature after model training, and $p_{pesudo}$ denotes the pseudo-label that will be obtained after further processing $p_m$, l represents the indicator function, and $ce$ represents the cross entropy loss.

$$L_{LMMD}=|{\displaystyle \frac{1}{N(N-1)}}\sum K_{xx}-{\displaystyle \frac{2}{N}}\sum K_{xi}+{\displaystyle \frac{1}{M(M-1)}}\sum K_{yy}-{\displaystyle \frac{2}{M}}\sum K_{yi}|$$

(13)

where M and N denote the number of weakly enhanced unlabeled and strongly enhanced unlabeled samples, respectively, $K_{XX}$ denotes the kernel matrix between strongly enhanced unlabeled, $K_{yy}$ denotes the kernel matrix between weakly enhanced unlabeled, $K_{xi}$ denotes the kernel matrix between strongly enhanced unlabeled and self samples, and $K_{yi}$ denotes the kernel matrix between weakly enhanced unlabeled and self samples.

2.3.3. Total Loss

$$L_{total}=L_s+\left({\mu }_{ce}·L_{ce}+{\mu }_{lmmd}·L_{LMMD}\right)$$

(14)

where $\left({\mu }_{ce}·L_{ce}+{\mu }_{lmmd}·L_{LMMD}\right)$ denotes the unsupervised loss $L_u$, ${\mu }_{ce}$ and ${\mu }_{lmmd}$ denote the weighting factors of $L_{ce}$ and $L_{lmmd}$, respectively.

Algorithm 1 is an optimization process for the overall loss of the system.

Algorithm 1 Algorithm of the Multi loss optimization

Input:  labeled data $X_l$, unlabeled data $X_u$, hyper-parameters $\alpha ,\beta$, loss weight ${\mu }_{ce},{\mu }_{lmmd}$, coefficient of noise $\zeta$ Output:  $L_{total}$   1: for  $i=1to100 \left(epoches\right)$  do   2:    $f_{unoise}^{\left(m\right)}$:$f_u^{\left(m\right)}+\zeta ·nois{e}^{\left(m\right)}$, where ${f_u}^{\left(m\right)}$ are features of unlabeled data $X_u$   3:    $L_s\leftarrow -log\left(p_k\right)-\alpha ·{\displaystyle \frac{{\displaystyle \sum _{i\ne k}}log\left(p_i\right)}{K-1}}+\beta ·{\displaystyle \frac{{\displaystyle \sum _{i\ne k}}{p}_{i}log\left(p_i\right)}{1-p_k}}$   4:    $L_{ce}\leftarrow {\displaystyle \frac{1}{M}}{\displaystyle \sum _{m=1}^M}1(max\left(p_m\right)>mask\left(argmax\left(p_m\right)\right)·ce({\widehat{p}}_m,p_{pesudo}))$   5:    $L_{LMMD}\leftarrow |{\displaystyle \frac{1}{N(N-1)}}\sum K_{xx}-{\displaystyle \frac{2}{N}}\sum K_{xi}+{\displaystyle \frac{1}{M(M-1)}}\sum K_{yy}-{\displaystyle \frac{2}{M}}\sum K_{yi}|$   6:    $L_{total}\leftarrow L_s+({\mu }_{ce}·L_{ce}+{\mu }_{lmmd}·L_{LMMD})$   7:    optimized by Adam   8: end for   9: return  $L_{total}$

2.3.4. Functions of Different Modules

The joint attention mechanism’s feature fusion technique enables the model to selectively concentrate on features that are most pertinent to the task at hand from various sources of features, uncovering the interrelationships among them. By assigning weights to the input features, it can visually display which aspects of the input features the model is inclined to focus on when making specific predictions, allowing for timely model adjustments.

The semi-supervised algorithm optimizes the performance of the model by combining supervised loss and unsupervised loss. Supervised loss directly guides the model to learn known label information, enabling the model to accurately classify on training data. Unsupervised loss helps models discover patterns and relationships in data by utilizing potential information from a large amount of unlabeled data, without relying on explicit labels. By minimizing both supervised and unsupervised losses simultaneously, the model achieves a balance between known and unknown. The model performs well on labeled data while also benefiting from unlabeled data, resulting in better generalization ability. Specifically, for supervised loss, adding the Jeffreys divergence to the cross entropy loss function maximizes the probability of the target class while reducing the model’s overconfidence in specific classes by smoothing the probability distribution of non target classes, thereby improving the model’s generalization ability. For unsupervised loss, LMMD loss measures the mean difference of local regions, allowing the model to focus on the local distribution of data and providing a more detailed explanation for the performance differences of the model in different data regions.

3. Experimental Section

3.1. Datasets

The Columbia-SRI-Colorado(CSC) [21] database is a lie detection corpus designed and recorded by researchers from Columbia University in 2006. The corpus includes 32 individuals, 16 males and 16 females, all speaking standard American English. Voice recording is divided into two stages, and participants can use lies to deceive the interviewer. In voice collection, the duration of each paragraph varies significantly depending on the type of question and the way it is answered, with some lasting 1 s and others exceeding 2–3 min. The speech sampling frequency of CSC library is 16 KHz, with a total duration of about 15.2 h. We obtained a total of 5412 voices (1 damaged) through text comparison, manual listening, and speech software cutting, with each voice lasting about 1–3 s. This article selected 5411 voices for the experiment.

3.2. Experimental Setup and Evaluation Metrics

This study investigated the practical performance of model algorithms with different numbers of labels, specifically set at 200, 600, and 1200 labels. Firstly, in the data augmentation section, different noise coefficients were selected for different types of features. For the 312-dimensional acoustic statistical features, the strong augmentation noise coefficient ${\lambda }_{austrong}$ was set to 1, and the weak augmentation noise coefficient ${\lambda }_{auweak}$ was set to 0.32. For the 64-dimensional log Mel spectrogram features, the strong augmentation noise coefficient ${\lambda }_{sustrong}$ was set to 1, and the weak augmentation noise coefficient ${\lambda }_{suweak}$ was set to 0.35.

All experiments were conducted on the Windows 11 operating system, utilizing an NVIDIA RTX 3060 graphics card (NVIDIA, Santa Clara, CA, USA). The experiments were performed using the PyTorch (Version: 2.2.1+cu118) deep learning framework and coded in the Python programming language. The Adam algorithm was employed for model optimization, with the number of iterations set to 100 epochs. In the model, the Dropout parameter was set to 0.9 to mitigate overfitting. Each experiment was repeated 10 times, and the average was computed to mitigate the influence of random errors.

The WA (Weighted Accuracy) is selected as the primary evaluation metric for assessing the performance of the recognition model.

In order to verify the rationality of the proposed model, three sets of ablation experiments were designed to analyze the improvement effect of the joint attention model and unsupervised fusion loss. The specific settings are as follows:

(1)

MFDF: Directly concatenate and fuse the 312 dimensional acoustic statistical features and 64 dimensional logarithmic Mel spectrogram features trained by the model in one dimension for lie detection.

(2)

MFDF + ML: Directly concatenate and fuse the 312 dimensional acoustic statistical features and 64-dimensional logarithmic Mel spectrogram features after model training, and combine them with a semi-supervised learning algorithm improved by LMMD loss for lie detection.

(3)

JAF: Using joint attention method to fuse the 312 dimensional acoustic statistical features and 64 dimensional logarithmic Mel spectrogram features after model training for lie detection.

As can be seen from

Table 1. Ablation experiment.

Model	Labeled Samples
	200	600	1200
MFDF	58.89	61.85	63.89
MFDF + ML	59.25	60.37	64.07
JAF	61.67	61.60	63.52
proposed algorithm	63.70	63.30	65.74

, the performance of the proposed algorithm is better than other modules. This is due to the fusion of the two types of features by joint attention, which further captures the detailed information of the two types of features and improves the lie detection accuracy to a certain extent; by introducing the semi-supervised joint optimization loss, the potential feature information is deeply mined, which effectively improves the lie detection accuracy. It is worth noting that the accuracy of 600 labels is lower than that of 200 labels. As the number of labels increases, more low-quality or noisy labels will be introduced, causing the model to be disturbed during prediction, thereby reducing accuracy. However, as the number of labels further increased to 1200, the model had sufficient data and information to better learn and distinguish subtle differences between different labels, resulting in an improvement in accuracy.

In Figure 3, we also show the accuracy and loss convergence curves of the algorithm in each group of experiments. To more clearly demonstrate the trend of changes, the number of iterations was set to 600. When the number of labels is 600, the experimental results indicate that the average accuracy of (d) is significantly higher than that of (a), (b), and (c), with the highest value reaching 63.3%. The highest values for (a) and (c) do not exceed 62%, and (b) performs the worst. Compared to (a) and (b), the loss for (c) and (d) has decreased. Overall, (d) demonstrates the best performance.

3.3. Comparative Experiments

To further verify the effectiveness of the proposed algorithm, two sets of comparative experiments were designed in this chapter. The specific settings are as follows:

(1)

SS-AE: Utilizes a semi-supervised auto encoder [22] network for speech lie detection. This network extracts latent representations of unlabeled data using an encoder and then reconstructs features using a decoder to compute the reconstruction loss between the original and reconstructed data. This loss is utilized as an unsupervised loss to optimize the model’s generalization ability on unlabeled data.

(2)

Mean-teacher: Based on the Mean-teacher model [12,23], which utilizes a combination of the reconstruction loss from the autoencoder network and the consistency regularization loss from the Mean-teacher model to optimize the decision boundary, enhancing the classification ability of the semi-supervised model.

From

Table 2. Comparative experiments of the same algorithm.

Model	Labeled Samples
	200	600	1200
SS-AE	58.37	56.94	59.87
Mean-teacher	55.45	58.39	60.84
proposed algorithm	63.70	63.30	65.74

, it can be observed that the proposed algorithm achieved accuracies of $63.70\%$, $63.30\%$, and $65.74\%$ for label counts of 200, 600, and 1200, respectively. These accuracies are significantly higher than those of the first comparative experiment SS-AE and the second comparative experiment Mean-teacher. These research findings indicate that the proposed method can learn the structure and features of unlabeled data and achieve better results.

In addition, we quantified the computational efficiency of the proposed algorithm. When the number of labels was set to 200, the total runtime of the model was 87.14 s, with an inference time of 6.22 s before the first iteration began. When the number of labels was set to 600, the total runtime was 204.33 s, and the inference time was 6.13 s. With 1200 labels, the total runtime increased to 402.52 s, with an inference time of 6.06 s. All experimental data are averages, and the results indicate that as the number of labels increases, the total training time of the model also increases, while the inference time shows no significant change.

3.4. Confusion Matrix

To further investigate the accuracy of lie detection, we introduced a confusion matrix to analyze the model on the CSC datasets. The confusion matrix, as shown in Figure 4, indicates that when the number of labeled data is 200, the recognition rates for truth and lies are $63\%$ and $64\%$, respectively; when the number of labeled data is 600, the recognition rates for truth and deception are $62\%$ and $63\%$, respectively; when the number of labels is 1200, the recognition rates for truth and lies are $65\%$ and $66\%$, respectively. It is noteworthy that the accuracy for lies consistently exceeds $60\%$.

4. Conclusions

In this paper, we propose a semi-supervised lie detection algorithm based on integrating multiple speech emotional features. First, acoustic statistical features and log Mel spectrogram features are extracted from the speech. A parallel network composed of an AE network and an LSTM network is then constructed to extract deep speech deception detection features. Subsequently, a joint attention method is proposed to enhance the fusion of features by leveraging the correlation between different features. Finally, a Local Maximum Mean Discrepancy (LMMD) loss is introduced in the unsupervised loss to help the model learn the intrinsic structure and feature information of unlabeled samples. A series of ablation experiments and comparative experiments are conducted on the CSC datasets to demonstrate the effectiveness of the proposed method in this paper.