Interactive Learning of a Dual Convolution Neural Network for Multi-Modal Action Recognition

Abstract

RGB and depth modalities contain more abundant and interactive information, and convolutional neural networks (ConvNets) based on multi-modal data have achieved successful progress in action recognition. Due to the limitation of a single stream, it is difficult to improve recognition performance by learning multi-modal interactive features. Inspired by the multi-stream learning mechanism and spatial-temporal information representation methods, we construct dynamic images by using the rank pooling method and design an interactive learning dual-ConvNet (ILD-ConvNet) with a multiplexer module to improve action recognition performance. Built on the rank pooling method, the constructed visual dynamic images can capture the spatial-temporal information from entire RGB videos. We extend this method to depth sequences to obtain more abundant multi-modal spatial-temporal information as the inputs of the ConvNets. In addition, we design a dual ILD-ConvNet with multiplexer modules to jointly learn the interactive features of two-stream from RGB and depth modalities. The proposed recognition framework has been tested on two benchmark multi-modal datasets—NTU RGB + D 120 and PKU-MMD. The proposed ILD-ConvNet with a temporal segmentation mechanism achieves an accuracy of 86.9% and 89.4% for Cross-Subject (C-Sub) and Cross-Setup (C-Set) on NTU RGB + D 120, 92.0% and 93.1% for Cross-Subject (C-Sub) and Cross-View (C-View) on PKU-MMD, which are comparable with the state of the art. The experimental results shown that our proposed ILD-ConvNet with a multiplexer module can extract interactive features from different modalities to enhance action recognition performance.

1. Introduction

As an important means of information and intelligent human–computer cooperation, action recognition has a broad application prospect in video monitoring, video retrieval, virtual reality, human–computer interaction, etc. [1,2,3,4,5]. Recently, deep convolutional neural networks (ConvNets) have strong feature learning and model generalization ability by automatically learning feature information from the bottom to the top, which have been applied to many fields, such as video understanding [6,7,8,9,10]. However, it is difficult to further improve the performance of models because of the limited ability of information representation and discrimination feature learning.

In recent years, most approaches mainly focus on designing deep ConvNets to learn spatial-temporal features, which improved action recognition performance through extracting the discriminative features from different modalities [11,12,13,14,15]. Most of those approaches are designed to work on a single frame or a single stack of frames in a short clip for extracting spatial-temporal features, while some methods obtain rich information by compressing action videos [16,17,18,19,20]. Although these ConvNets based methods have achieved amazing performance in action recognition, the performance improvement of the models is limited due to the inherent limitations, e.g., some video frame sparse sampling methods are easy to cause the loss of temporal information and most single-stream ConvNets cannot jointly learn interactive features of different modalities.

Inspired by the video compression representation method proposed in [17,21,22], in this paper, we use the compression method to generate visual dynamic images (VDIs) and depth dynamic images (DDIs) to obtain rich information from those two modalities, and a multi-modal feature interactive learning model (ILD-ConvNet) is provided to extract the interactive features of RGB and depth modalities in order to improve recognition performance.

The main contributions of this paper are as follows:

• An effective 2D-ConvNet framework is proposed to extract interactive features from different modalities to recognize human action, which jointly learn features from different modalities for human action recognition.
• We experimentally demonstrate that the constructed VDIs and DDIs can effectively represent spatial-temporal information from the whole RGB-D sequences.
• The proposed ILD-ConvNet recognition framework has been evaluated on large-scale human action datasets. The results show that our proposed ILD-ConvNet achieved better recognition performance and generalization performance, which demonstrated the effectiveness of our ILD-ConvNet.

2. Related Work

The extraction of underlying features is mainly divided into spatial and temporal features as for RGB and depth sequences, which can well describe the details of human actions. Early low-level feature information extraction mainly includes feature extraction algorithms with prior knowledge, geometric features, motion information, and Histogram of Oriented Gradient (HOG). [23,24]. In addition, some recognition methods used to represent time information mainly focus on spatial configuration modeling, dynamic time axis (DTW), Hidden Markov model (HMM), and the recognition methods based on hand-draft are summarized in [25,26].

In the past two decades, with the development of deep learning networks, action recognition methods based on RGB and depth modalities have made great progress. Simonyan et al. proposed a two-stream ConvNet recognition framework to fuse RGB and optical flow features, which achieved the effect of combining spatial-temporal information [27]. Wang et al. [28] present a series of good training practices for deep action recognition, Temporal Segment Networks (TSN), which included a sparse temporal sampling strategy and video-level supervision. However, it is difficult to ensure that the interactive information between different streams can be differentiated and explored. Zhang et al. [29] designed a cross-stream ConvNet to investigate syndicated information from multiple modalities. Wang et al. [30] provided a Temporal Difference Network (TDN) to learn multi-scale temporal information. In addition, 3D ConvNets can obtain more comprehensive spatiotemporal feature information, such as C3D, I3D, and X3D [31,32,33]. However, the amount of computation is too large and requires a lot of computing resources. In addition, transformer-based methods achieved remarkable results due to globally connected patterns across spatial and temporal dimensions [34,35,36].

Recently, some researchers have designed a series of ConvNets to realize human action recognition based on depth sequences, because it can reflect the comprehensive three-dimensional geometric information. Wang et al. [37,38] expressed the depth sequences to dynamic images through rank pooling and applied 3D point cloud mapping transformation to improve recognition performance, which achieved relatively good recognition results on multi-modal datasets. In addition, Recurrent Neural Networks (RNNs) can capture the feature information of temporal well, which have developed rapidly in action recognition [39,40,41].

The above feature learning methods have achieved success in improving the human action recognition performance. However, the single-modality data cannot provide the comprehensive information required for action recognition. Based on the multi-stream recognition framework, some strategies are proposed to capture distinguishing features, such as sparse temporal sampling [14], motion image [33] and dynamic image representation [17,21,22]. In [42], a cooperative ConvNets is designed to jointly learn the middle-level feature information of RGB and depth sequences, which obtained good recognition performance on the existing RGB-D datasets such as NTU RGB + D 60 [43]. In addition, the joint learning mechanisms are designed to learn single-modal and cross-modal information for action recognition tasks, such as c-ConvNet [42], SC-ConvNets [44] and J-ResNet-CMCB [45]. In addition, some approaches improved recognition performance through fusing RGB, depth and skeleton in multi-stream feature learning, such as Modality Compensation Network [46] and multi-modal action recognition model [47].

To overcome the lack of the above multi-stream ConvNets, with the advent of an increasing amount of multi-modal data, we are motivated to design more efficient recognition frameworks to represent spatial-temporal information from entire action sequences and extract complimentary features from different modalities.

3. The Proposed Method

The proposed ILD-ConvNet architecture as shown in Figure 1, which has three parts: the ILD-ConvNet, network input representation and joint optimization.

3.1. ILD-ConvNet Framework

By considering 2D ConvNets, 3D ConvNets and RNNs, a dual ConvNets framework is designed based on the multi-modal feature interactive learning and fusion strategy. The designed multi-modal interactive learning dual flow network ILD-ConvNet consists of two separate paths, multiplexer modules and two losses. Given an RGB and depth sequences, the rank pooling method converted them into a pair of dynamic images $\langle VDI,DDI\rangle$. Then, a designed single ResNet with multiplexer modules is used to extract the discriminative features from $\langle VDI,DDI\rangle$. Then, the two losses and effective fusion strategy are used to jointly optimize and fuse multi-modal features.

Due to the high modularity, ResNet is selected as the basic module for RGB and depth information flow learning [48]. The ILD-ConvNet recognition framework includes two independent information flow learning networks. Each network includes $7\times 7$ convolution layer, four bottleneck blocks, a full connection layer and softmax mapping layer. The network adopts the same strategies as ResNet. The network framework based on the ResNet₅₀ model is summarized in

Table 1. The network architecture of ILD-ConvNet-based ResNet₅₀. Down sampling is performed by conv3_x, conv4_x and conv5_x with a stride of 2.

Layer Name	Output Size	ILD-ConvNet50
		RGB Stream	Depth Stream
Conv1	$112\times 112$	$14\times 14$, 64, stride 2	$7\times 7$, 64, stride 2
Multiplexer_1	$112\times 112$	$\left[\begin{array}{c}1\times 1, 64\times 2\\ 1\times 1, 64\end{array}\right]\times 1$
Conv2_x	$56\times 56$	$\left[\begin{array}{c}1\times 1, 64\\ 3\times 3, 64\\ 1\times 1, 256\end{array}\right]\times 3$	$\left[\begin{array}{c}1\times 1, 64\\ 3\times 3, 64\\ 1\times 1, 256\end{array}\right]\times 3$
Multiplexer_2	$56\times 56$	$\left[\begin{array}{c}1\times 1, 256\times 2\\ 1\times 1, 256\end{array}\right]\times 1$
Conv3_x	$28\times 28$	$\left[\begin{array}{c}1\times 1, 128\\ 3\times 3, 128\\ 1\times 1, 512\end{array}\right]\times 4$	$\left[\begin{array}{c}1\times 1, 128\\ 3\times 3, 128\\ 1\times 1, 512\end{array}\right]\times 4$
Multiplexer_3	$28\times 28$	$\left[\begin{array}{c}1\times 1, 512\times 2\\ 1\times 1, 512\end{array}\right]\times 1$
Conv4_x	$14\times 14$	$\left[\begin{array}{c}1\times 1, 256\\ 3\times 3, 256\\ 1\times 1, 1024\end{array}\right]\times 23$	$\left[\begin{array}{c}1\times 1, 256\\ 3\times 3, 256\\ 1\times 1, 1024\end{array}\right]\times 23$
Multiplexer_4	$14\times 14$	$\left[\begin{array}{c}1\times 1,1024\times 2\\ 1\times 1,1024\end{array}\right]\times 1$
Conv5_x	$7\times 7$	$\left[\begin{array}{c}1\times 1, 512\\ 3\times 3, 512\\ 1\times 1, 2048\end{array}\right]\times 3$	$\left[\begin{array}{c}1\times 1, 512\\ 3\times 3, 512\\ 1\times 1, 2048\end{array}\right]\times 3$
Multiplexer_5	$7\times 7$	$\left[\begin{array}{c}1\times 1, 2048\times 2\\ 1\times 1, 2048\end{array}\right]\times 1$
FC	$1\times 1$	FC-RGB	FC-depth

. In the ResNet₅₀ model, the input maps of the 4 bottleneck blocks are 64, 256, 512 and 1024. To reduce the number of parameters of the multiplexer module, the network is placed before each bottleneck block to learn the interaction characteristics of different modalities.

3.2. Network Inputs

Represent RGB or depth sequences as $\langle I_1,I_2,I_3,\dots ,I_T\rangle$. $\varnothing \left(I_t\right)\in R^d$ is the feature vector extracted therefrom of $I_t$, and the average feature vector at the time t is defined as

$$V_t=\frac{1}{t}{\sum }_{\tau =1}^t\varnothing (I_{\tau }),$$

(1)

where the sorting score function of $V_t$ at the time t is as follows.

$$S\left(t|d\right)=\langle \mathrm{d},V_t\rangle ,\mathrm{d}\in R^d,$$

(2)

where d is the learned parameter vector. The score $S\left(t|d\right)$ reflects the ordering of each image frame in the sequence, which satisfies $q>t\to S\left(q|d\right)>S\left(t|d\right)$.

$$E\left(d\right)=\frac{\lambda }{2}{\left|\left|d\right|\right|}^2+\frac{2}{T\left(T-1\right)}{{\displaystyle \sum }}_{q>t}\max \left\{0,1-S\left(q|d\right)+S\left(t|d\right)\right\},$$

(3)

where the first and second terms are the quadratic term of SVM optimization and the error accumulation of score ranking, respectively. The interval between q and t is set as 1 to distinguish the score ranking. The mapping function from video to parameter feature d* vector is further defined as [49]:

$$d^{*}=\rho \left(I_1,\dots ,I_T;\varnothing \right)=argmi{n}_{d}E\left(d\right),$$

(4)

where d* is the optimal estimate of $d$. $d^{*}$ contains all the ordering information of the video sequence, which is equal to the number of pixels of a single frame of the sequence. Therefore, it can be seen as a descriptor of the videos and interpreted as a standard image frame. For depth sequence, we first normalize them into single-channel images, and then use the introduced rank pooling method ($d^{*}$) to compress them into one dynamic image (DDI).

3.3. Multiplexer Modules

In terms of network architecture design, ResNet is selected as the basic network based on the highly modular residual unit. The proposed ILD-ConvNet is divided into two paths to process the input RGB and depth information flow characteristics, respectively. High-level interactive learning is conducted through the multiplexer module, as shown in Figure 2.

The multiplexer module combines the characteristics of two-path information flow. After weighted redirection interactive learning, it is input back to RGB and depth information flow, respectively. The characteristic output function of the multiplexer module can be defined as

$$F_{multiplexer}\left(x\right)=ReLU\left(BN\left(f_{1\times 1}\left(ReLU\left(BN\left(f_{1\times 1}\left(w\ast x\right)\right)\right)\right)\right)\right),$$

(5)

where $BN(·)$ and $ReLU(·)$ denote the batch normalization and rectified linear units activation function, respectively, and $f_{1\times 1}$ is a learnable $1\times 1$ convolution kernel. The information flow input is defined as:

$$X_{RGB}=\lambda X_{RGB}+\left(1-\lambda \right)F_{multiplexer}\left(X_{RGB}+X_{depth}\right),$$

(6)

$$X_{depth}=\lambda X_{depth}+\left(1-\lambda \right)F_{multiplexer}\left(X_{RGB}+X_{depth}\right),$$

(7)

where $X_{RGB}$ and $X_{depth}$ represent the output of the information flow, respectively. $\mathsf{\lambda }\in \left(0,1\right)$ is the adjustment coefficient.

3.4. Optimization Learning

Different from the ConvNets, which is trained on a single frame, the designed ILD-ConvNet has paired sample objects $\langle VDI,DDI\rangle$ for training. In order to eliminate the deviation between two different modalities, a dual loss function is used to jointly optimize RGB and depth information flows. In the meantime, standard cross-entropy loss functions are used in the training process.

The information flow of the training sample $\langle VDI,DDI\rangle$ is expressed as $X_{\langle VDI,DDI\rangle }$. The scores of $VDI$ and $DDI$ are obtained by using the softmax function. The classification probability score can be estimated as follows:

$$pro{b}_c=\frac{\exp (W_{c}X+b_c)}{{\displaystyle {\sum }_{c_i}\exp (w_{c_i}X+b_{c_i})}},$$

(8)

where $W_c$ and $b_c$ are the weights and offsets of the softmax layer. The loss function of the two streams can be described as:

$$L_{RGB}(y,C)=-{\displaystyle \sum _{c=1}^C{y}_c(\log (pro{b}_{c-RGB}))},$$

(9)

$$L_{depth}(y,C)=-{\displaystyle \sum _{c=1}^C{y}_c(\log (pro{b}_{c-depth}))},$$

(10)

where $C$ is the classification number of actions, $y_c$ is the ground truth label, $pro{b}_{c-RGB}$ and $pro{b}_{c-depth}$ are the classification probabilities, respectively.

Different from training on a single RGB or depth frame, ILD-ConvNet operates on $\langle VDI,DDI\rangle$ and obtains two-stream features. To make the proposed ConvNets more discriminative and eliminate the multi-modal variance between the RGB-D information streams, a dual loss function is adopted to jointly optimized ILD-ConvNet. In the training process, the standard cross-entropy loss is used to jointly optimize ILD-ConvNet. The overall loss function can be expressed as:

$$L(y,C)=L_{RGB}+L_{depth},$$

(11)

The feature vectors $v_{RGB}$ and $v_{depth}$ of the dual stream network can be obtained in the testing process. The final score feature vectors of RGB and depth sequences can be obtained through feature fusion:

$$v_{fusion}=v_{RGB}◦v_{\mathrm{depth}},$$

(12)

where “$◦$” represents the maximum, minimum or inner product operation. Then, the maximum value of $v_{fusion}$ is used as the classification result of RGB and depth sequences.

3.5. Datasets Implementation Details

3.5.1. Datasets

NTU RGB + D 120 [50] is the largest multi-modal human action dataset currently released. NTU RGB + D 120 includes RGB, depth, infrared IR and 3D skeleton modalities, and two evaluation protocols are defined. For the Cross-Subject (C-Sub) evaluation protocol, samples collected by subjects with camera numbers of 1, 2, 4, 5, 8, 9, 13, 14, 15, 16, 17, 18, 19, 25, 27, 28, 31, 34, 35, 38, 45, 46, 47, 49, 50, 52, 53, 54, 55, 56, 57, 58, 59, 70, 74, 78, 80, 81, 82, 83, 84, 85, 86, 89, 91, 92, 93, 94, 95, 97, 98, 100, and 103 were used for training (63,360 samples). The rest were selected for testing (51,120 samples). For the Cross-Setup (C-Set), the samples collected by the even ID cameras are divided into training (54,720 samples) and the rest for testing (59,760 samples).

The PKU-MMD dataset [51] includes multi-modal data such as RGB, depth, infrared radiation and skeleton modalities, and each modality includes 1074 long video sequences, and two action recognition evaluation protocols are designed for the dataset. For the Cross-Subject (C-Sub) evaluation protocol, it is divided into 18,134 training and 2600 testing samples. For the Cross-View (C-View) evaluation protocol, the training set have 13,815 samples and the testing have 6919 samples.

3.5.2. Implementation Details

Network input. The RGB and depth sequences are compressed to obtain the input of the ILD-ConvNet by adopting the rank pooling method proposed above. In order to adapt to the dual flow network training model, $\langle VDI,DDI\rangle$ is scaled to $224\times 224$. The DDI sorted by depth sequence is scaled to $\left[0,255\right]$. In addition, the training samples are incrementally processed by random clipping and dithering.

Model training and testing. The parameters of ILD-ConvNet can be initialized through the pre-trained weights of ResNet. In the training and testing process, we follow the good practice proposed in [16], the methods of mini batch, momentum, random gradient descent and data augmentation methods are used for feature learning, where the batch and momentum are set as 64 and 0.8, respectively. The initial learning rate is set as 0.01, which decreases to 0.1 times every 30 generations. In order to overcome the overfitting phenomenon, dropout technology was further adopted. The dropout rate was set to 0.5. Inspired by the good practices proposed in [16], we further verified the impact of segmentation strategy on recognition performance in our experiments.

4. Experimental Results and Analysis

4.1. Effectiveness of the Proposed ILD-ConvNet

4.1.1. Effectiveness

In order to evaluate the effectiveness of the proposed recognition framework, comparative experiments of different modalities are conducted on the NTU RGB + D 120 dataset, including RGB, depth and RGB + depth dual modalities. The parameter $\lambda$ was set to 0.1 and the average fusion method is used in the experiments,

Table 2. The results of ResNet₅₀ and ILD-ConvNet.

Methods	Modality	C-Sub	C-Set
ResNet50	RGB	40.6%	39.5%
ResNet50	depth	43.6%	42.2%
ResNet50	RGB + depth	51.4%	51.3%
ResNet50	$VDI$	56.7%	52.5%
ResNet50	$DDI$	72.2%	72.3%
ResNet50	$\langle VDI,DDI\rangle$	73.8%	74.0%
ILD-ConvNet50	$\langle RGB,depth\rangle$	55.4%	55.0%
ILD-ConvNet50	$\langle VDI,DDI\rangle$	75.3%	75.2%

shows the experimental results. The recognition rate of VDI in ResNet₅₀ model is 16.1% and 13.0% higher than that of RGB, and the DDI is 28.6% and 30.1% higher than that of depth on the basis of the evaluation protocols of the NTU RGB + D 120 dataset C-Sub and C-Set. The improvement of the recognition rate shows that the proposed video frame ranking method can improve the behavior recognition performance of the training mode. The results proves that this method can effectively represent the spatial-temporal information of samples.

Table 2 shows that the recognition rates of ResNet-50 model achieve 51.4% and 51.3%, respectively, by fusing RGB and depth modalities. The recognition rates of ILD-ConvNet₅₀ achieve 55.4% and 55.0%, respectively. The recognition rates of ILD-ConvNet50 are 4.0% and 3.7% higher than those of ResNet₅₀. The recognition rates of ResNet₅₀ model achieve 73.8% and 74.0% by fusing VDI and DDI modalities. ILD-ConvNet₅₀ can obtain 75.3% and 75.2% recognition rates. The proposed ILD-ConvNet₅₀ are 1.5% and 1.2% higher than those of ResNet₅₀. Furthermore, we can see that the recognition rate is greatly improved by fusing RGB and depth modalities, which indicates the interactive characteristics between RGB and depth sequences. Furthermore, the experimental results of our ILD-ConvNet show that learning the interactive characteristics between two different modalities can better improve recognition performance. The reason of the improvement of recognition rate is that the designed multiplexer module has learned the interaction characteristics of RGB and depth.

4.1.2. Comparison of Different Fusion Methods

The average fusion method was adopted to achieve the classification results in Table 2. To obtain better recognition performance, the weighted average, maximum and product score fusion methods are evaluated on the NTU RGB + D dataset, and the results are shown in

Table 3. Comparative accuracies of the proposed ILD-ConvNet₅₀ with $\langle VDI,DDI\rangle$ on the NTU RGB + D 120 dataset.

Methods	C-Sub	C-Set
Average	75.3%	75.2%
Product	75.2%	75.0%
Max	74.9%	74.9%

. It can be seen from the experimental results that the recognition rates of the three feature fusion methods are similar, and the average method is relatively good. Therefore, the average fusion method is selected as the feature fusion method for the following experiments.

4.1.3. Effectiveness of Parameter $\lambda$

The parameter $\lambda$ is varied from 0 to 0.5 to evaluate recognition performance. The comparison results are summarized in Figure 3, and we can see that increasing $\lambda$ will lead to worse performance. The results implies that increasing $\lambda$ does not help obtain richer information from the other modality. Thus, $\lambda$ = 0.1 is set in the following experiments.

4.2. Comparison to the State of the Art

4.2.1. Experimental on the NTU RGB + D Dataset

To further demonstrate the effectiveness of RGB and depth modalities, BNInception, ResNet₅₀, ResNet₁₀₁, ILD-ConvNet₅₀ and ILD-ConvNet₁₀₁ were selected for further experiments, and the comparison results are shown in

Table 4. Comparative accuracies of different networks, and the segment is set 3 in TSN [16].

Methods	Modality	C-Sub	C-Set
BNInception	$\langle RGB,depth\rangle$	52.5%	53.1%
ResNet50	$\langle RGB,depth\rangle$	51.4%	51.3%
ResNet101	$\langle RGB,depth\rangle$	56.5%	54.1%
BNInception	$\langle VDI,DDI\rangle$	75.4%	76.2%
ResNet50	$\langle VDI,DDI\rangle$	73.8%	74.0%
ResNet101	$\langle VDI,DDI\rangle$	77.2%	78.4%
ILD-ConvNet50	$\langle RGB,depth\rangle$	55.4%	55.0%
ILD-ConvNet101	$\langle RGB,depth\rangle$	57.2%	56.2%
ILD-ConvNet50	$\langle VDI,DDI\rangle$	75.3%	75.2%
ILD-ConvNet101	$\langle VDI,DDI\rangle$	82.4%	83.1%
J-ResNet-CMCB [45]	$\langle VDI,DDI\rangle$	82.8%	83.6%
TSN [16] + SC-ConvNet [44]	$\langle VDI,DDI\rangle$	86.9%	87.7%
TSN [16] + ILD-ConvNet101	$\langle VDI,DDI\rangle$	86.9%	89.4%

. RGB and depth modalities are used as inputs in this group of experiments. In addition, BNInception, ResNet₅₀ and ResNet₁₀₁ are used to train RGB, depth, $VDI$ and $DDI$, respectively. Meanwhile, image pairs $\langle RGB,depth\rangle$ and $\langle VDI,DDI\rangle$ are used as the inputs of ILD-ConvNet₅₀ and ILD-ConvNet₁₀₁.

The experimental results indicate that ILD-ConvNet₁₀₁ achieved 82.8% and 83.6% recognition rates, respectively, on the C-Sub and C-Set protocols of the NTU RGB + D 120 dataset. ILD-ConvNet₁₀₁ is 0.7% and 2.1% higher than ResNet₁₀₁, respectively, for the same input and base network, and our ILD-ConvNet₁₀₁ is 5.6% and 6.2% higher than ResNet₁₀₁, respectively. Furthermore, inspired by the segmentation training mechanism proposed in [16], we conduct experiments in 3 segments on the NTU RGB + D 120 dataset, which obtain 86.9% and 89.4% on two protocols. In specially, the proposed ILD-ConvNet degenerates to the plain SC-ConvNet as in [44]. Although the proposed recognition framework is only 1.7% higher than SC-ConVnet in C-Set protocol, it also shows the effectiveness of our designed multiplexer module. Moreover, some subtle actions are easier to recognize, such as “jump table tennis ball”. The experimental results demonstrated that the interactive features of RGB and depth modalities learned by the designed multiplexer module can improve recognition performance.

4.2.2. Experimental on the PKU-MMD Dataset

To evaluate the effectiveness and generalization ability of the proposed ILD-ConvNet, we further conducted experiments on the multi-modal human action dataset PKU-MMD with different scales and application backgrounds. The basic network is ResNet₁₀₁, and $\lambda$ is set to 0.1.

Table 5. Comparative accuracies of different networks on the PKU-MMD dataset; the segment is set 3 in TSN [16].

Methods	Modality	C-Sub	C-View
SA-LSTM [52]	skeleton	86.3%	91.4%
TA-LSTM [52]	skeleton	86.6%	92.3%
STA-LSTM [52]	skeleton	86.9%	92.6%
ResNet101	$VDI$	77.3%	74.8%
ResNet101	$DDI$	81.7%	79.0%
ResNet101	$\langle VDI,DDI\rangle$	83.6%	82.7%
ILD-ConvNet101	RGB + depth	80.7%	79.8%
J-ResNet-CMCB [45]	$\langle VDI,DDI\rangle$	90.4%	91.4%
TSN [16] + SC-ConvNet [44]	$\langle VDI,DDI\rangle$	92.1%	93.2%
TSN [16] + ILD-ConvNet101	$\langle VDI,DDI\rangle$	92.0%	93.1%

shows the experimental comparison results.

Table 5 shows that the proposed ILD-ConvNet based on ResNet₁₀₁ obtained 92.0% and 93.1% recognition rates, respectively, on the two protocols. The results show that the proposed ILD-ConvNet obtained generalization ability on different types and scales of datasets.

4.3. Analysis

Compared with other ConvNet-based methods, the experimental results on the NTU RGB + D and PKU-MMD datasets have shown the effectiveness of our ILD-ConvNet. It can be seen that the proposed dual-stream framework can achieve better recognition performance. The experimental results also implied that the introduced rank pooling mechanism can represent spatial-temporal information from entire RGB and depth sequences, and the designed multiplexer module can enhance recognition performance by extracting the interactive features from different modalities.

Figure 4 and Figure 5 shows the confusion matrix under the two evaluation protocols. It can be seen that the designed ILD-ConvNet can effectively identify most of the actions, but some refined actions, such as “make an OK sign” and “snap fingers”. On the C-Sub test protocol, highly similar actions of human–object interaction are easy to be confused, such as “put on glasses” and “take off glasses”, “reading” and “writing”. In addition, it is difficult to effectively distinguish actions with similar behavior trajectories, such as “rub two hands” and “clasping”.

Figure 6 and Figure 7 are the confusion matrix under the two test protocols in the PKU-MMD dataset. The PKU-MMD is a long video sequence multi-modal dataset. Each video sequence contains a series of actions, which needs to be preceded by segmentation. We can see from Figure 6 and Figure 7 that the proposed ILD-ConvNet can distinguish most human actions after segmentation. However, some actions cannot be effectively identified, such as “eat meal/snack”, “drink water”, “tear up paper” and “put on glasses”.

The hyper parameter $\lambda$ controls the weights of the previous convolution layers in ILD-ConvNet. Adjusting $\lambda$ should improve the recognition performance of the proposed ILD-ConvNet. We first roughly get the range of $\lambda$ through analyzing the contribution of different item in Equations (6) and (7). The higher $\lambda$ forces the RGB and depth streams to obtain more interactive features from different modalities. However, more noise will be introduced at the same time, which will affect the recognition performance of the proposed network model. Therefore, we should select the optimal $\lambda$ through verifying the results in practice.

5. Conclusions

A human action recognition method based on interactive learning of multi-modal features is proposed in this paper. The rank pooling method is used to construct $\langle VDI,DDI\rangle$ to obtain spatial-temporal information, and a dual-stream framework, ILD-ConvNet, with multiplexer modules is then employed to learn the interactive features from the RGB-D modalities for improving recognition performance. The experimental results illustrate the effectiveness of the rank pooling method and a multiplexer module. Compared with the state-of-the-art methods, our proposed recognition framework achieves comparable results on two multi-modal RGB-D datasets. The future research of this work is to extend the 2D ConvNet framework to 3D, and explore more effective feature fusion methods and reduce the number of model parameters.