This chapter will experimentally verify the effects of the attention branch and micro-attention branch we propose. The experiment involves three directions: (a) Does the application of micro-attention branches generate improvements compared to the original network structure? (b) When the module is a multi-branch structure, is it reasonable to directly increase the attention unit? (c) In the same network structure, is there a correlation between the number of micro-attention branches and the classification effect?
4.1. Experimental Setup
Because the proposed attention branch acts on the 3D convolution module in the neural network, we chose the excellent I3D network [
3] as the basic network for the experiment. Its network structure is shown in
Figure 4. There are multiple basic modules with similar structures in this framework. The structure of these basic modules is shown in
Figure 5A, and we named them Module A. The module consists of four independent sub-branches (from right to left in the figure, named Branch1 to Branch4). Existing papers have proved that the network structure has achieved excellent results on multiple video datasets.
According to the content in
Section 3.2, we modified the basic module (as shown in
Figure 5) of I3D. We added attention branches and different numbers of micro-attention branches respectively to construct 3 different modules, named Module B (
Figure 5B: using two micro-attention units, which are added to the two sub-branch of the original module), Module C (
Figure 5C: using a micro-attention unit, which is only used on a sub-branch of the original module), and Module D (
Figure 5D: the original module is considered as a whole, and an attention branch is added). Taking the first basic module in the basic I3D network as an example, we introduce the specific structure of these four modules (the structure is shown in
Table 3). Branch1 to 4 are the four sub-branches of the basic module A; ModuleAtt represents the attention branch structure directly added with module A as a whole; Branch1Att and Branch2Att represent the micro-attention branches added by Branch1 and Branch2, respectively.
On the premise of keeping the basic network structure unchanged, these four modules are used to replace the basic modules in the original network I3D to construct four networks for training and testing. Network 1 is the original network, called I3D; Network 2 is a network that uses module B as the basic module, called ; and Network 3 is a network that uses module C as the basic module, called .
The experimental training and test data used Kinetics [
38], which is the open source video dataset. We selected seven action categories (exercising with an exercise ball, parasailing, washing dishes, pull ups, cleaning shoes, folding paper, pumping fist) from it. The number of video clips in the dynamics dataset is shown in
Table 4.
Data augmentation is known to be of crucial importance for the performance of deep architectures. We process the selected videos through the CV tool, and extract 80 frames with 224∗224 resolution from each video as the input of the network. For shorter videos, we looped the video as many times as necessary to satisfy each model’s input interface.
The above networks are independently trained from scratch in the same software and hardware environment. The machine-learning library we use is tensorflow, version V1.5. The programming language we use is python, version v3.6. Training on videos used standard ’stochastic gradient descent’ with momentum set to 0.9 in all cases. Models were trained using a similar learning rate.
4.2. Results and Discussion
Table 5 reports the results and the monitoring content of these networks during the training process. Comparing the first two rows of
Table 5, we can find interesting results. It can be found that the
network which uses two micro-attention branches obtains the best results. Compared with I3D, the classification accuracy of
is improved by 3.6%, while the number of parameters is increased by only 0.6%. We believe that this is due to the reasonable integration of the attention features extracted by the attention branch and the features extracted by the original module. Then, we compare the data in rows 1 and 3 of
Table 5. The accuracy of the
network is also increased by 2.8% compared to I3D. We can add micro-attention branches in the network to improve the accuracy of the model in dealing with action-recognition tasks.
Then, we compare the data in rows 2 and 3 of
Table 5. It can be found that the recognition accuracy of the
network with two micro-attention branches and the
network with one micro-attention branch is higher. It can be seen that when there are multiple sub-branches in the basic module of the network, more micro-attention branches are added in the module, and the higher the accuracy of the network model for action recognition. That is, the number of micro-attention branches is directly proportional to the effect of the final model.
Figure 6 shows the accuracy of each model under different action categories. We can find that
produces consistently better results than other network on all the classes. We believe that this is the result of the more effective integration of attention features into the process of feature extraction. In addition, we can find that in the classification of ‘washing dishes’ and ‘cleaning shoes’, the results of the
network are only slightly improved compared with the original network. This is because the action subject is obvious in this kind of video, and there is almost no complex background. In the classification of ‘pumping fist’, due to the complex background of this kind of video, the recognition accuracy of
is significantly higher than that of the original network.
Finally, we compare the time consumption of an iteration. We can find that the training of the I3D network takes the longest time. We believe that this is due to the structural characteristics of the residual network, which speed up the training speed. Through this structure, adding the attention branch to the original module can improve the adjustment efficiency of the parameters of the feedforward network.
Comparison with scarecrow model. We still choose I3D as the basic network and add attention branches by taking the basic modules as a whole. The compared network that uses module D as the basic module is called RA3D.
It can be found from
Table 6 that although the parameter amount of RA3D increased by 25.0% compared to
, the classification accuracy decreased. The effect of the RA3D network is not only worse than the
network, but also worse than the original I3D network. This shows that when there are multiple branches in the network module, directly using the attention branch will increase the resource consumption in the training phase and negatively affect the accuracy of the model. It can be seen that the micro-attention branch is suitable for application in a multi-branch network, and can improve the recognition accuracy of the original network while not increasing resource consumption as much.
By comparing the contents in
Table 5 and
Table 6, we can find that among the four networks, the network that uses the micro-attention branches has the highest recognition accuracy. This is because in the network module of multiple sub-branches, different branches have different emphases on extracting features, so these sub-branches cannot be taken as a whole. This makes it impossible to use 3D attention branches directly for this type of module. In the test phase, the testing time of the networks with added attention branches is longer than that of the original network. This is because these new networks add a small number of operations such as convolution. The increase is within the acceptable range.
During the experiment, we recorded the changes in memory occupation and time consumption of four network structures during training (the relevant content is shown in
Figure 7). Combining the content in
Table 5 and
Table 6, we can find the following phenomena: Although the average memory occupation of the ra3d network is the highest (8.7 GB) and the time-consumption is the longest (72,917 s), its classification effect is the worst. In the training process of I3D, the average memory occupation is the smallest (7.5 GB), but the network time consumption (57,924 s) is slightly higher than that of the network with micro-attention branches. This shows that adding micro-attention branches through the residual network structure can improve the training efficiency of the model.
Through the above experiments, it can be found that the attention branch has a positive effect on the model. However, when there are multiple convolution sub-branches in the stacking module of the network, the attention branch cannot be used directly. At this time, micro-attention branches can be added independently to multiple sub-branches of the original network, which can effectively improve the classification effect of the final model and reduce the training time. Under the same conditions, the number of micro-attention units has a positive effect on the training effect.
Comparison with other networks. We choose to compare our results with other attention algorithms (2D + LSTM). ResNET [
41] is selected as the 2D Network in this structure. The last layer of multiple ResNETs is connected with LSTM, and video frames are input to different ResNETs according to time sequence. The number of ResNets in the structure is related to the number of video frames. This means that with the increase in pictures, the number of ResNETs also increases. In order to control the number of parameters, we limit the input to 10 frames instead of 80 frames. We evenly select 10 frames from the video clips.
It can be found from
Table 7 that although we deliberately reduced the number of input images, the parameters of the 2D + LSTM network increased by 46.81% compared with
. We also found that its memory consumption was much higher than that of
. However, the accuracy of 2D + LSTM is only two-thirds that of of
. This shows that although 2D + LSTM can theoretically obtain features of the spatiotemporal dimension, it only uses high-level features and cannot capture the primary features of video frames. Our proposed attention branch can be added anywhere in the network, so it can capture features of different levels.
Applied to two-stream 3D convolution network. Although 3D ConvNet should be able to learn motion features directly from RGB input, existing networks prefer to make full use of multi-source data in video, such as optical flow. Therefore, we apply a 3D micro-attention branch to a two-stream I3D network to test the effect of the structure. We choose the two-stream network which is state-of-the-art, with one network trained on RGB inputs and another trained on optical flow inputs which carry optimized, smooth flow information. We used the
network instead of I3D to construct a comparative experiment. The two branches of the network have exactly the same structure. Video frames are processed in the same way as other networks. We computed optical flow with a TV-L1 algorithm [
42]. We trained the two networks separately and averaged their predictions at test time.
It can be found from
Table 8 that the accuracy of two-stream network with the micro-attention branch is improved compared with the original network. 3D attention branches can be used not only to extract attention features from video, but also from optical flow. We believe that the application conditions of micro-attention branches are only related to the network structure.