Depression Detection in Speech Using Transformer and Parallel Convolutional Neural Networks

Abstract

As a common mental disorder, depression becomes a major threat to human health and may even heavily influence one’s daily life. Considering this background, it is necessary to investigate strategies for automatically detecting depression, especially through the audio modality represented by speech segments, mainly due to the efficient latent information included in speech when describing depression. However, most of the existing works focus on stacking deep networks in audio-based depression detection, which may lead to insufficient knowledge for representing depression in speech. In this regard, we propose a deep learning model based on a parallel convolutional neural network and a transformer in order to mine effective information with an acceptable complexity. The proposed approach consists of a parallel convolutional neural network (parallel-CNN) module used to focus on local knowledge, while a transformer module is employed as the other parallel stream to perceive temporal sequential information using linear attention mechanisms with kernel functions. Then, we performed experiments on two datasets of Distress Analysis Interview Corpus-Wizard of OZ (DAIC-WOZ) and Multi-modal Open Dataset for Mental-disorder Analysis (MODMA). The experimental results indicate that the proposed approach achieves a better performance compared with the state-of-the-art strategies.

1. Introduction

The mental disorder of depression affects a huge number of people in the world [1,2], making them suffer from psychological pain and pessimism, or even leading to an increasing tendency of self-abuse and suicide [3,4]. Depression itself is difficult to diagnose empirically, since it mainly depends on the patient’s self-assessment [5] and other evaluation questionnaires [6,7], which may largely be affected by subjective factors. Hence, the design of effective depression detection systems with an objective standard of diagnosis is needed for the purpose of reducing these subjective effects [8].

Fortunately, the current research on automatic depression detection provides feasible methods for this scenario, including video-based [9,10,11], text-based [12,13], audio- based [5,14], electroencephalogram (EEG)-based [15,16], and multi-modal fusion [17,18,19,20,21,22] strategies. Within these strategies, speech in audio signals has been proved as effective in detecting depression due to the latent depression factors hidden in speech [23,24]. To design a speech-based depression detection system, existing research considers deep models employ stacking convolutional neural networks (CNNs) or recurrent neural network (RNN) structures [25,26,27]. This may lead to insufficient depression-related information obtained by the output layers due to the single-stream networks [28,29], and may also result in a large time complexity and memory footprint [30]. In addition, previous works on speech emotion recognition (SER) show the effectiveness in mining latent emotional information from multi-category features in speech [31,32,33], which implies the possibility of including parallel structures for the current depression detection task through learning from multiple representations [34].

In view of the limitations in the existing works, we propose a convolutional neural network based on a parallel structure and transformer model in this paper in order to learn effective depression representations through multi-source fusion with acceptable resource occupation. First, we set low-level mel-frequency cepstral coefficient (MFCC) features with a fixed length as the input to the network. Then, a parallel CNN structure was employed to process these inputs for the purpose of perceiving local knowledge. Afterwards, we added an improved transformer with a linear attention mechanism as a third stream of the proposed approach, instead of conventional RNNs, in order to capture temporal sequential information in speech. The linear attention mechanism utilizes the kernel function in constructing the inner product instead of the original scaled dot-product attention, in order to reduce the complexity of the attention mechanisms. Note that, compared with existing related works on depression detection in speech, the proposed approach utilizes CNN streams to learn local knowledge while setting a transformer stream for acquiring global sequential information.

The main contribution of this paper can be summarized as follows:

• We propose a transformer-CNN-CNN (TCC) model for depression detection in speech, with two modules of a parallel CNN and transformer.
• Within the proposed TCC, we employed linear attention mechanisms instead of the original attention in multi-head attention mechanisms to reduce time complexity.
• Within the proposed TCC, a fusion layer was then added to merge the outputs from the CNNs and transformer to obtain sufficient local and global information for fusion decisions.

The remainder of this paper is organized as follows. Section 2 introduces highly related works for the proposed approach, while Section 3 shows the proposed model consisting of a parallel CNN and a transformer based on a parallel structure, including the improvement scheme of the attention mechanisms. Then, Section 4 represents the databases, experimental setups, and analysis of the experimental results. The conclusion of this paper is provided in Section 5.

2. Related Works

2.1. Depression Detection in Speech

As a typical human psychological disorder, depression requires an accurate and fast diagnosis through speech communication [23,35], inspiring the works on automatic depression detection in speech [24,36,37]. The initial research on speech-based depression detection depends on shallow-structure algorithms of support vector machines (SVMs) [38] and Gaussian mixture models (GMMs) [39]. Then, Liu et al. [40] proposed a novel decision tree for depression detection. The emergence of deep learning makes it possible to utilize deep neural networks for performing more accurate depression detection in speech [14,34,41,42]. On exploring the theories for speech-based depression detection, Dubagunta et al. found that neurophysiological changes can occur during depression, which may affect laryngeal control (i.e., the behavior of the vocal folds) [43]. Along this direction, they used a CNN-based structure to demonstrate the effectiveness of depression detection only using the audio modality of speech [43]. Then, in this way, Zhao et al. also explored employing a structure using a CNN and long short-term memory (LSTM) with a multi-head attention mechanism [31], yet the inclusion of recurrent units may incur a high computational complexity [30]. In addition to the research on exploring learning algorithms, handcrafted features are also proved as effective when confronting speech-based depression detection [26,44].

2.2. Transformer

The transformer model proposed by Vaswani et al. shows a better performance and efficient parallelization in the field of natural language processing (NLP) compared with the conventional RNN and CNN-based networks [30]. Seeing the success of the transformer, following works also support its success in the fields of image, speech, and multi-modal processing [45,46,47,48]. Within the related works, a spatial transformer was proposed for image compositing [49]. Then, Chen et al. proposed a pre-trained image processing transformer that performs well in adapting to different image processing tasks [50]. In the field of speech-related works, Dong et al. expanded the transformer to speech recognition, which achieves a low word error rate and faster training [51]. Wang et al. presented an end-to-end model based on stacked transformer layers for SER tasks, which outperformed the prior arts by a large margin, still making use of the CNN-RNN-based architecture [52]. Related works on transformer-based SER also prove the feasibility of using transformer structures in computational paralinguistics [53,54].

2.3. Attention Mechanisms

Attention mechanisms initially achieved great success in the field of image processing and NLP [55,56,57]. In speech processing, Mirsamadi et al. introduced local attention based on a recurrent neural network for automatic speech recognition (ASR) [58], while Xie et al. and Jiang et al. utilized multiple attention mechanisms to achieve a better performance for SER tasks [59,60]. Regarding the transformer, multi-head attention is proposed for parallel computing, applied to the fields of image [61], language [62], speech [31,63], and multi-modal [64] processing, allowing the model to focus on the information of the representations from different locations. However, due to the computational complexity required for softmax calculation, the attention mechanism algorithm requires squared time-memory complexity with the input length [65,66]. Thus, in order to reduce the computational complexity of the attention mechanisms, Shen el al. proposed an efficient attention module through applying softmax functions to the query and key vectors, considering the association of matrix multiplication [67], while Tsai et al. proposed a kernel-based attention module through mathematical analysis on the attention-calculation formula [68].

3. The Proposed Transformer-CNN-CNN

The overall structure of the proposed approach is shown in Figure 1. For the convenience of narration, we use transformer-CNN-CNN to refer to the proposed model, containing two modules of a parallel CNN and transformer. The parallel-CNN module consisted of three convolutional layers with pooling operations, and the transformer module included four transformer layers with multi-head attention, both utilizing the inputs from 40-dimensional low-level MFCC extraction [69], considering the usage of MFCC in existing speech-based depression detection works [70,71]. The outputs of the two modules were then processed through flattening (only for the parallel-CNN module) and concatenating, prior to feeding to the softmax layer to make a final decision.

3.1. The Parallel-CNN Module

Inspired by conventional CNN models for image and sequential signal processing (e.g., LeNet, AlexNet, and VGGNet) [72,73], the CNN architecture used in this paper set three stacking convolutional layers for each CNN stream, with 16, 32, and 64 channels, respectively. Using the parallel CNN’s representations, the mel spectrogram X was processed as a grayscale image, with the two dimensions of time and frequency, noted as ‘$T\times d$’, where the d mel-spectrogram point was set to 40. The size of the convolutional kernel for each convolutional layer was set to 3, whereas the stride step was 1. Each of the convolutional layers activated by a rectified linear unit (ReLU) function was followed by batch normalization (BN). Then, we performed maximum pooling and employed dropout operations afterwards for each convolutional layer with a rate of $0.3$ in order to reduce the influence from overfitting.

3.2. The Transformer Module

Different from the original transformer, we improved the transformer structure in this paper and only a similarity-encoder structure was used in order to adapt to the current task of detecting depression in speech. In view of the existing research on transformers, a deep model does not always perform well when stacking a large number of transformer layers [74,75]. Hence, the transformer module of the proposed TCC contained four stacked transformer layers as a transformer encoder, including a multi-head attention sublayer and a feed-forward sublayer for each transformer layer. We further added residual skip connections for each sublayer, as shown in Figure 2.

Let the input of the lth transformer layer be $X_l^{\left(T\right)}\in {\Re }^{T_0\times d}$, where $T_0$ is the length of the processed temporal sequence using $4\times 1$ maximum pooling (see Figure 1) on the time dimension of the initial input X. Hence, the output of the corresponding multi-head attention sublayer is represented as

$$X_l^{(T,MA)}=X_l^{\left(T\right)}+MA\left(X_l^{\left(T\right)}\right),$$

(1)

where $MA(·)$ is the mapping function for the multi-head attention. Thus, the output of the feed-forward sublayer can be calculated as

$$X_l^{(T,FF)}=X_l^{(T,MA)}+ReLU(X_l^{(T,MA)}·W_1+b_1)·W_2+b_2,$$

(2)

where $ReLU(·)$ represents an ReLU activation, while the linear mappings are $W_1\in {\Re }^{d\times d_0}$ and $W_2\in {\Re }^{d_0\times d}$ with there corresponding biases $b_1$ and $b_2$, using $d_0=512$ nodes for the hidden layer, with a dropout operation setting its dropout rate to $0.4$.

3.2.1. Multi-Head Attention

Then, we focused on the multi-head attention unit as shown in Figure 2, which endows a model with sufficient information through parallel training. Within this unit, an input vector was divided into multiple spaces, and a self-attentional mechanism was employed to allow for a model’s parallel training while capturing sufficient target knowledge. Compared with the current conventional single-head attention mechanism [76], multi-head attention can improve the effective resolution, thus enhancing the representation of speech using multiple spaces while reducing interference from noise through the average-pooling operator.

In a multi-head attention unit from the lth transformer layer, we first present the calculation process of query, key, and value vectors as $Q_{l,n^{\left(S\right)}}=X_l^{\left(T\right)}·W_{Q_{l,n^{\left(S\right)}}}$, $K_{l,n^{\left(S\right)}}=X_l^{\left(T\right)}·W_{K_{l,n^{\left(S\right)}}}$, and $V_{l,n^{\left(S\right)}}=X_l^{\left(T\right)}·W_{V_{l,n^{\left(S\right)}}}$, respectively, with $n^{\left(S\right)}=1,2,\dots ,N^{\left(S\right)}$. Note that the $W_{Q_{l,n^{\left(S\right)}}}\in {\Re }^{d\times d_K}$, $W_{K_{l,n^{\left(S\right)}}}\in {\Re }^{d\times d_K}$, and $W_{V_{l,n^{\left(S\right)}}}\in {\Re }^{d\times d_V}$ represent the corresponding linear-mapping matrices for the three sorts of vectors, where we set $d_K=d_V=\lfloor \frac{d}{N^{\left(S\right)}}\rfloor$.

Afterwards, we performed concatenating (noted as ‘$Concat(·)$’) considering the outputs from all of the heads as

$$MA\left(X_l^{\left(T\right)}\right)=Concat(H_{l,1},H_{l,2},\dots ,H_{l,N^{\left(S\right)}}),$$

(3)

with the output of the $n^{\left(S\right)}$th head as

$$H_{l,n^{\left(S\right)}}=Atten(Q_{l,n^{\left(S\right)}},K_{l,n^{\left(S\right)}},V_{l,n^{\left(S\right)}}),$$

(4)

where $Atten(·)$ represents the operator from an attention unit.

3.2.2. The Attention Unit

First, we employed scaled dot-product attention in the attention unit based on single-head attention mechanisms [30,76]. The main idea of the scaled dot-product attention is to enhance the representation of the current time step by introducing contextual information. For the $n^{\left(S\right)}$th head from the lth transformer layer, the query $Q_{l,n^{\left(S\right)}}$ represents the content of interest, while the key $K_{l,n^{\left(S\right)}}$ is equivalent to the tags of all of the words in the current dictionary. The influence degree of the context for the current time step is reflected through the dot-product result of $Q_{l,n^{\left(S\right)}}$ and $K_{l,n^{\left(S\right)}}$. Then, the relevance measure was obtained by softmax normalization, which was used to weight the value $V_{l,n^{\left(S\right)}}$ in order to obtain the attention score.

Hence, the calculation process of the scaled dot-product attention can be represented as

$$Atten(Q_{l,n^{\left(S\right)}},K_{l,n^{\left(S\right)}},V_{l,n^{\left(S\right)}})=Softmax\left(\frac{Q_{l,n^{\left(S\right)}}·K_{l,n^{\left(S\right)}}^T}{\sqrt{d_K}}\right)·V_{l,n^{\left(S\right)}},$$

(5)

in which $Softmax(·)$ represents the softmax function.

Further, by applying the softmax function to

$$\left\{\begin{array}{cc}{Q}_{l,n^{\left(S\right)}}={[q_{l,n^{\left(S\right)},1},q_{l,n^{\left(S\right)},2},\dots ,q_{l,n^{\left(S\right)},T_0}]}^T\in {\Re }^{T_0\times d_K},\hfill & \\ K_{l,n^{\left(S\right)}}={[k_{l,n^{\left(S\right)},1},k_{l,n^{\left(S\right)},2},\dots ,k_{l,n^{\left(S\right)},T_0}]}^T\in {\Re }^{T_0\times d_K},\hfill & \\ V_{l,n^{\left(S\right)}}={[v_{l,n^{\left(S\right)},1},v_{l,n^{\left(S\right)},2},\dots ,v_{l,n^{\left(S\right)},T_0}]}^T\in {\Re }^{T_0\times d_V},\hfill \end{array}\right.$$

(6)

we obtained the ith row ($i=1,2,\dots ,T_0$) of the attention unit’s output as

$$\begin{array}{cc}Atte{n}_i(Q_{l,n^{\left(S\right)}},K_{l,n^{\left(S\right)}},V_{l,n^{\left(S\right)}})\hfill & \\ =\frac{[e^{\frac{q_{l,n^{\left(S\right)},i}^T{k}_{l,n^{\left(S\right)},1}}{\sqrt{d_K}}},e^{\frac{q_{l,n^{\left(S\right)},i}^T{k}_{l,n^{\left(S\right)},2}}{\sqrt{d_K}}},\dots ,e^{\frac{q_{l,n^{\left(S\right)},i}^T{k}_{l,n^{\left(S\right)},T_0}}{\sqrt{d_K}}}]}{{\sum }_{j=1}^{T_0}{e}^{\frac{q_{l,n^{\left(S\right)},i}^T{k}_{l,n^{\left(S\right)},j}}{\sqrt{d_K}}}}·V_{l,n^{\left(S\right)}},\hfill \end{array}$$

(7)

resulting in the form of

$$Atte{n}_i(Q_{l,n^{\left(S\right)}},K_{l,n^{\left(S\right)}},V_{l,n^{\left(S\right)}})=\frac{{\sum }_{m=1}^{T_0}\left(e^{\frac{q_{l,n^{\left(S\right)},i}^T{k}_{l,n^{\left(S\right)},m}}{\sqrt{d_K}}}{v}_{l,n^{\left(S\right)},m}^T\right)}{{\sum }_{j=1}^{T_0}{e}^{\frac{q_{l,n^{\left(S\right)},i}^T{k}_{l,n^{\left(S\right)},j}}{\sqrt{d_K}}}}.$$

(8)

Then, considering the similarity measurement of the scaled dot-product attention [77], we further rewrote Equation (8) as

$$Atte{n}_i(Q_{l,n^{\left(S\right)}},K_{l,n^{\left(S\right)}},V_{l,n^{\left(S\right)}})=\frac{{\sum }_{m=1}^{T_0}\left(Sim(q_{l,n^{\left(S\right)},i},k_{l,n^{\left(S\right)},m})v_{l,n^{\left(S\right)},m}^T\right)}{{\sum }_{j=1}^{T_0}Sim(q_{l,n^{\left(S\right)},i},k_{l,n^{\left(S\right)},j})},$$

(9)

where $Sim(·)$ represents the similarity measurement between two arguments of vectors. In view of the possible $O\left(T_0^2\right)$ computational complexity for the softmax calculation in processing long sequences, we further set non-negative mappings and inner products in order to replace the softmax for representing the similarity measurement [68]. Hence, we induced two non-negative mappings $\varphi (·)$ and $\phi (·)$ to perform linear inner products for $q_{l,n^{\left(S\right)},i}$ and $k_{l,n^{\left(S\right)},j}$ (or $k_{l,n^{\left(S\right)},m}$), respectively, leading to $Sim(q_{l,n^{\left(S\right)},i},k_{l,n^{\left(S\right)},j})={\varphi }^T\left(q_{l,n^{\left(S\right)},i}\right)\phi \left(k_{l,n^{\left(S\right)},j}\right)$. Employing this similarity form, we obtained

$$Atte{n}_i(Q_{l,n^{\left(S\right)}},K_{l,n^{\left(S\right)}},V_{l,n^{\left(S\right)}})=\frac{{\sum }_{m=1}^{T_0}\left({\varphi }^T\left(q_{l,n^{\left(S\right)},i}\right)\phi \left(k_{l,n^{\left(S\right)},m}\right)v_{l,n^{\left(S\right)},m}^T\right)}{{\sum }_{j=1}^{T_0}{\varphi }^T\left(q_{l,n^{\left(S\right)},i}\right)\phi \left(k_{l,n^{\left(S\right)},j}\right)},$$

(10)

which was further transformed into

$$Atte{n}_i(Q_{l,n^{\left(S\right)}},K_{l,n^{\left(S\right)}},V_{l,n^{\left(S\right)}})=\frac{{\varphi }^T\left(q_{l,n^{\left(S\right)},i}\right){\sum }_{m=1}^{T_0}\left(\phi \left(k_{l,n^{\left(S\right)},m}\right)v_{l,n^{\left(S\right)},m}^T\right)}{{\varphi }^T\left(q_{l,n^{\left(S\right)},i}\right){\sum }_{j=1}^{T_0}\phi \left(k_{l,n^{\left(S\right)},j}\right)}.$$

(11)

When assuming that, for the matrix forms,

$$\phi \left(K_{l,n^{\left(S\right)}}\right)={[\phi \left(k_{l,n^{\left(S\right)},1}\right),\phi \left(k_{l,n^{\left(S\right)},2}\right),\dots ,\phi \left(k_{l,n^{\left(S\right)},T_0}\right)]}^T\in {\Re }^{T_0\times d_K},$$

(12)

we rewrote the ith row of the attention unit’s output as

$$Atte{n}_i(Q_{l,n^{\left(S\right)}},K_{l,n^{\left(S\right)}},V_{l,n^{\left(S\right)}})=\frac{{\varphi }^T\left(q_{l,n^{\left(S\right)},i}\right)\left({\phi }^T\left(K_{l,n^{\left(S\right)}}\right)·V_{l,n^{\left(S\right)}}\right)}{{\varphi }^T\left(q_{l,n^{\left(S\right)},i}\right)\left({\phi }^T\left(K_{l,n^{\left(S\right)}}\right)\mathbf{e}\right)},$$

(13)

where all of the elements $\mathbf{e}\in {\Re }^{T_0\times 1}$ are equal to one. Hence, the computational complexity changes into $O\left(T_0{\left(d_K\right)}^2\right)$. For both of the non-negative mappings $\varphi (·)$ and $\phi (·)$, we employed the exponential linear unit (ELU) activation as in [68].

4. Experiments

4.1. Experimental Setups

We employed the datasets of Distress Analysis Interview Corpus-Wizard of Oz (DAIC-WOZ) [78] and Multi-modal Open Dataset for Mental-disorder Analysis (MODMA) [43,79] in this paper. The setup of one-time training-test splitting was included in the experiments in accordance with existing related research [31,80,81].

The DAIC-WOZ dataset was composed of interview recordings for the clinical diagnosis of psychological stress, conducted by Ellie, an animated virtual interviewer, with a sampling rate of 16 kHz. The dataset included 42 randomly selected patients with depression and 47 healthy speakers. In order to ensure the validity of the sample, we eliminated the abnormal speech segments containing a small amount of information—that is, less than 3 s and more than 12 s—leading to a total of 4401 samples (2156 depression and 2245 normal). Then, 1000 samples were randomly selected as the test set, whereas the rest of the samples formed the training part, considering the balance of data distribution between the two classes as in Ref. [31]. The MODMA dataset contained 52 subjects, including 23 subjects (7 females; 16–55 years) with depression and 29 healthy-control subjects (9 females; 18–55 years). The speakers were recorded by interviewing, reading, and picture description. After segmenting the original speech, the average duration for the segments exceeded 10 s. Hence, a total of 1321 samples were obtained through eliminating the abnormal samples from the data (590 depression and 731 normal), in which, we randomly selected 350 samples as the test set. Note that, despite the partially speaker-dependent cases in splitting the sets, we employed the random splitting, since the unsupervised side information (e.g., speaker, text, duration, and noise) provides relatively low impact on the supervised depression-related information.

Within the experiment, we employed the Hanning window, with a window length of 25 ms in speech framing, and a window shift of 10 ms. The speech segments with a length of 9 s were intercepted by the Python library of Librosa to extract the MFCC features [82]. The network employed a batch size of 32 using the adaptive moment estimation (Adam) optimizer, with the initial learning rate equal to $0.0005$.

4.2. Experimental Results

4.2.1. Experiments for TCC

First, for the purpose of verifying the validity of the multi-stream structure within the proposed model, we make a comparison on the depression-detection performance between a single CNN, parallel CNN, transformer, and the proposed TCC model (using the scaled dot-product attention). For the TCC model, its parameter settings keep consistent with the parallel CNN, and the multi-head numbers used in the transformer and TCC are both set to 4. In this regard, we present Figure 3 in order to show the unweighted accuracy (UA) or equivalently unweighted average recall (UAR) convergence curves of the four models on the two datasets, respectively.

It is learnt from the figure that the proposed TCC model achieves the best UA performance, followed by the parallel CNN, transformer, and finally the single-CNN model. Among them, the TCC performs better compared with the parallel CNN, while there is a slight gap between the transformer and the two models. According to the performance of the parallel CNN, we still believe that the CNN has a strong feature extraction ability and can learn effective feature representation from almost any input to complete the classification task. The transformer is inferior in feature extraction compared to the parallel CNN, yet it can be implemented using paralleled computing so as to improve the training efficiency of the model. As for the proposed TCC model, the multi-head attention mechanism in the transformer can strengthen a simple CNN model’s ability to capture temporal features by dividing the input into multiple spaces for processing. Considering the analysis and experimental results, the proposed TCC model may be optimal among these models in comparison.

To further investigate the optimal performance of the proposed model, we changed the number of the attention heads $N^{\left(S\right)}$ used in the experiments for the two datasets. The models can be represented as ‘TCC-Single’ and ‘TCC-$N^{\left(S\right)}$’, respectively, where TCC-Single represents the single-head attention mechanism used by the model and TCC-$N^{\left(S\right)}$ indicates that the model utilizes a multi-head attention mechanism. The attention unit in TCC-Single and TCC-$N^{\left(S\right)}$ was set to scaled dot-product attention. We directly trained the models on the two datasets, and used UAs to evaluate the effectiveness of the models, as shown in

Table 1. The unweighted accuracies (%) of the proposed TCC approach with different numbers of heads in the multi-head attention mechanism (TCC-Single, TCC-2, TCC-4, TCC-8, and TCC-16).

Models∖Datasets	DAIC-WOZ	MODMA
TCC-Single	$84.5$	$89.0$
TCC-2	$90.4$	95.8
TCC-4	93.8	$94.6$
TCC-8	$91.5$	$93.8$
TCC-16	$91.0$	$91.5$

. It can be seen from the table that the TCC-$N^{\left(S\right)}$ models achieve better UA results compared with the TCC-Single model on the two datasets. Specifically, TCC-4 achieves the best UA performance ($93.8\%$) on the DAIC-WOZ dataset, whereas TCC-2 performs the best ($95.8\%$) on the MODMA dataset.

In view of the experimental results in Table 1, we chose the number of heads in the multi-head attention as $N^{\left(S\right)}=4$ in the following experiments. Then, in order to compare both the accuracy and time-complexity performance of different attention computing units more precisely, we only used the transformer stream without the parallel-CNN streams, with the UA convergence curves shown in Figure 4 and Figure 5, with respect to iteration steps and the training time (within 3000 iteration steps) on the two datasets. We employed the two setups of scaled dot-product attention (noted as ‘softmax’) and linear inner-product attention (noted as ‘kernel’ due to the similar mapping assumption as in kernel methods [83]). In Figure 4, we show the UAs starting from 400 iteration steps, and then present the results every 200 steps. Similarly, in Figure 5, we set 80 s as the start point for presenting the UAs, also using 200 steps as the interval. It can be seen that the UA performance of the proposed model when using the linear attention is similar to that when using the scaled dot-product attention, and even slightly better compared with the scaled dot-product attention. In addition, the ‘kernel’ setup leads to a lower training time compared with the ‘softmax’ setup due to the reduced complexity in the matrix calculation.

Finally, we present the confusion matrices for the proposed TCC (including TCC-softmax and TCC-kernel) on DAIC-WOZ and MODMA datasets, considering the experimental results of the metrics, as shown in Figure 6. The results indicate that the proposed TCC performs well in classifying both normal and depression speech. In addition, it is observed that the TCC-kernel outperforms TCC-softmax on the MODMA dataset and is insignificant for the UAs of the two setups on both of the datasets (at the significance level of $0.05$ using a one-tailed z-test); hence, it is feasible to use the TCC-kernel directly due to its lower computational complexity.

4.2.2. Experimental Comparisons

Afterwards, in order to prove the validity of the proposed depression detection approach, we present an experimental comparison between the proposed TCC approach (with ‘softmax’ and ‘kernel’ setups) and the state-of-the-art models of 2D-CNN-LSTM [84], CNN-bi-directional LSTM (BLSTM) using end-to-end (E2E) multi-task learning (MTL) with self-attention (noted as ‘CNN-BLSTM+E2EMTL+SA’) [27], the speech-based depression detection approach decision tree (DT) [40], and DepAudioNet [85], with the same parametric setups as in the corresponding works for the compared existing approaches. Note that all of these approaches in the comparison share the same experimental settings and data inputs. The evaluation metrics of precision, recall, and F1-score for the positive class (depression) were considered in the result comparison, with the corresponding experimental results shown in

Table 2. The precisions, recalls, and F1-scores (%) of the proposed TCC approach (including ‘softmax’ and ‘kernel’ setups) and the state-of-the-art approaches (2D-CNN-LSTM, CNN-BLSTM+E2EMTL+SA, DT, and DepAudioNet) on the DAIC-WOZ and MODMA datasets.

Approaches∖Metrics	Precision	Recall	F1-Score
DAIC-WOZ Dataset:
2D-CNN-LSTM [84]	$91.2$	$92.0$	$91.6$
CNN-BLSTM+E2EMTL+SA [27]	$82.2$	$89.1$	$85.5$
DT [40]	$78.7$	$80.0$	$79.2$
DepAudioNet [85]	$82.4$	$84.0$	$83.2$
TCC-softmax (Proposed)	95.0	92.2	93.6
TCC-kernel (Proposed)	$94.3$	$90.6$	$92.4$
MODMA Dataset:
2D-CNN-LSTM [84]	$92.9$	$93.5$	$93.2$
CNN-BLSTM+E2EMTL+SA [27]	$93.5$	$90.1$	$91.8$
DT [40]	$79.6$	$81.3$	$80.4$
DepAudioNet [85]	$83.9$	$87.6$	$85.7$
TCC-softmax (Proposed)	$96.0$	$92.3$	$94.1$
TCC-kernel (Proposed)	98.7	94.8	96.7

. Note that the precision and recall metrics can be acquired based on true and predicted positive labels, thus obtaining the F1-score through the harmonic average for the precision and recall. As can be learned from Table 2, the proposed approach exceeds the model of LSTM architectures (see 2D-CNN-LSTM, CNN-BLSTM+E2EMTL+SA, and DepAudioNet) in all of the metrics. For the DAIC-WOZ dataset, the best F1-score of the proposed TCC model is $93.6\%$, whereas, for the MODMA dataset, the TCC-kernel achieves the best F1-score of $96.7\%$, which is $2.6\%$ higher than TCC-softmax.

To make further comparisons, we present the average results of these metrics as shown in

Table 3. The (unweighted) average precisions, recalls (UA), and F1-scores (macro) (%) of the proposed TCC approach (including ‘softmax’ and ‘kernel’ setups) and the state-of-the-art approaches (2D-CNN-LSTM, CNN-BLSTM+E2EMTL+SA, DT, and DepAudioNet) on the DAIC-WOZ dataset.

Approaches∖Metrics (Average)	Precision	Recall (UA)	F1-Score (Macro)
DAIC-WOZ Dataset:
2D-CNN-LSTM [84]	$90.8$	$92.6$	$91.7$
CNN-BLSTM+E2EMTL+SA [27]	$85.3$	$89.5$	$87.3$
DT [40]	$78.9$	$80.2$	$79.6$
DepAudioNet [85]	$82.8$	$84.9$	$83.8$
TCC-softmax (Proposed)	93.9	93.8	93.8
TCC-kernel (Proposed)	$92.8$	$92.7$	$92.7$

. Note that the average results in Table 3 refer to the average values of the precision, recall (equivalent to UA), and F1-score across the two classes in depression detection. It is proved that the proposed TCC approach is effective for the prediction of depression and that the proposed TCC approach significantly outperforms the existing CNN-BLSTM+E2EMTL+SA, DT, and DepAudioNet approaches (at the significance level of $0.01$ using the one-tailed z-test) when using the UA (or UAR) metric. Specifically, the TCC-softmax setup significantly outperforms these approaches at the significance level of $0.001$. Therefore, through the experimental results and analysis, it is proved that the proposed TCC approach performs well for depression detection in speech.

5. Conclusions

This paper proposed a transformer-CNN-CNN (TCC) approach for depression detection in speech. The proposed approach employed a parallel convolutional neural network (CNN) and a transformer using a parallel structure, resulting in three parallel streams to obtain local and temporal information, considering linear attention to reduce the complexity of the model. Then, we evaluated the proposed approach on the DAIC-WOZ and MODMA datasets, proving the effectiveness for the proposed TCC approach. The future works will focus on two aspects. First, it will be possible to explore effective acoustic low-level features in presenting depression in speech. Then, further parallel structures based on the proposed TCC can be investigated in order to improve the detection performance through parallel computing.