Attention Modulated Multiple Object Tracking with Motion Enhancement and Dual Correlation

Abstract

The one-shot multiple object tracking (MOT) framework has drawn more and more attention in the MOT research community due to its advantage in inference speed. However, the tracking accuracy of current one-shot approaches could lead to an inferior performance compared with their two-stage counterparts. The reasons are two-fold: one is that motion information is often neglected due to the single-image input. The other is that detection and re-identification (ReID) are two different tasks with different focuses. Joining detection and re-identification at the training stage could lead to a suboptimal performance. To alleviate the above limitations, we propose a one-shot network named Motion and Correlation-Multiple Object Tracking (MAC-MOT). MAC-MOT introduces a motion enhance attention module (MEA) and a dual correlation attention module (DCA). MEA performs differences on adjacent feature maps which enhances the motion-related features while suppressing irrelevant information. The DCA module focuses on decoupling the detection task and re-identification task to strike a balance and reduce the competition between these two tasks. Moreover, symmetry is a core design idea in our proposed framework which is reflected in Siamese-based deep learning backbone networks, the input of dual stream images, as well as a dual correlation attention module. Our proposed approach is evaluated on the popular multiple object tracking benchmarks MOT16 and MOT17. We demonstrate that the proposed MAC-MOT can achieve a better performance than the baseline state of the arts (SOTAs).

1. Introduction

Multiple object tracking (MOT) aims at estimating the locations of multiple objects and providing identifications. The MOT plays a fundamental role in many applications, such as intelligent driving [1], human–computer interaction, and pedestrian behavior analysis [2,3].

Currently, MOT can be summarized into two paradigms: two-stage and one-shot frameworks. The two-stage framework follows the tracking-by-detection paradigm [4,5,6,7], which divides MOT into two separate tasks: (i) object detection in terms of getting the objects bounding boxes, and (ii) object feature extraction and association in terms of finding the objects’ tracking traces across the time course. At the detection stage, convolution neural network (CNN) detectors such as Yolov3 [8] and Faster-RCNN [9] are applied to localize all objects of interest in the images by a number of bounding boxes. Then, these images are cropped and resized into the same resolution. At the feature extraction and association stage, data association across frames is conducted based on the re-identification.

In this two-stage approach, earlier literature works use color histogram [10,11,12] and pixel-based template representation [13,14] features to describe the objects. Lately, feature extraction based on CNNs has been popular. There have been many works with different structures, such as the Siamese network [15], which are capable of learning discriminative features cropped from the detected object bounding boxes. Some works use the appearance-based cost function while others pursue more complicated association mechanisms, such as motion and topology cues. This group of the methods [16,17,18] has achieved promising results on the public benchmark datasets. Finally, the Hungarian algorithm [19] is used to optimize the cost function to accomplish the optimal association results. Although effective in performance, the two-stage paradigm is time-consuming [20]. This is because the representation model needs to extract the ReID features from each individual bounding box. Furthermore, the two-stage paradigm depends on the performance of the detector.

On the other hand, the one-shot paradigm has drawn more attention in the MOT research community, with its maturity of multi-task learning in deep learning. The one-shot model integrates the detection and representation tasks into a unified system. This would reduce the inference time, which makes the system more real-time. However, the tracking accuracy of the one-shot paradigm is not as high as in the two-stage paradigm. The one-shot approach combines the detection and re-identification, which overlooks the specificity and commonality of different tasks in learning and the assistance role of motion information. The one-shot approach is capable of providing the target location and its associated embedding feature simultaneously, derived from a shared feature map. Nevertheless, there is an issue regarding the design. This is that two tasks of detection and re-identification are trained at the same time, using the same image input. The inherent differences between the detection and the re-identification tasks are overlooked. This creates a competition due to the different optimization processes. Specifically, the loss of the detection task is to narrow the feature distance between the different objects, while that of the re-identification is the opposite. This would result in learning a suboptimal solution, which degrades the performance. Additionally, the other problem is that current one-shot paradigms have a lack of motion information. Motion information plays an important role in understanding the objects’ movement. The object motion is often different from the background movement, due to the camera motion. Hence, a lack of a motion modulated model would miss the critical motion information for the objects.

To alleviate the above issues of the one-shot paradigm, we propose a novel Motion and Correlation-Multiple Object Tracking (MAC-MOT) model to improve the performance of the one-shot tracking. First, symmetry plays an essential role in designing our proposed framework. The details are explained in the following: (i) The backbone is a Siamese-based deep learning network, where two branches of the Siamese are symmetric in design. (ii) The input of dual stream images has two input branches, where images are fed in consecutive order into the system in a symmetric manner. (iii) A dual correlation attention module, which is proposed to balance the two output streams of the proposed framework. All of these reflect the idea of symmetry.

Second, the MAC-MOT framework has the following components. Initially, it takes adjacent frames as input and uses the multi-scale convolution network to extract features, respectively. We propose a motion enhance attention module (MEA), which utilizes motion information as a guide to focus on important features. MEA exploits global average pooling (GAP) to squeeze the feature maps to only focus on the channel-level importance. Subsequently, it uses the temporal difference as an approximate motion map. To solve the competition between detection and ReID tasks, we introduce a dual correlation attention module (DCA) to decouple the input. Inspired by the Dual Attention Network (DANet) [21], we use the self-attention mechanism to obtain the task-related features. Meanwhile, we use the correlation mechanism to protect the commonality of features.

In summary, the main contribution of our work is two-fold:

(1)

We propose a motion enhance attention module (MEA) to model the motion-relate feature, which obtains the weight for the feature channel.

(2)

We introduce a dual correlation attention module (DCA) in order to reduce ambiguity in learning different tasks. DCA makes the one-shot method more adaptive to multi-task-based representation learning.

This paper is presented in the following structure. First, related work is introduced in Section 2. The proposed model is explained in Section 3. Experimentation and results are discussed in Section 4. Finally, we conclude this paper in Section 5.

2. Related Work

The two-stage method focuses on individual tasks at each stage. The individual stage can focus on a single task without risking other tasks’ performance. However, the two-stage method is usually slow because both object detection and re-identification feature embedding need an inference effort. For this reason, the two-stage method cannot be used in many practical applications.

In order to overcome the shortcomings of the two-stage method, the one-shot method integrates the detection and re-identification into a unified framework, which simultaneously outputs the bounding box and the embedding feature. They can share the weight of the backbone in different tasks to reduce the time of inferencing. In detail, the one-shot framework consists of three parts, the feature extractor, the detection branch, and the re-identification branch. For instance, the Joint Detection Embedding (JDE) [20] approach models the training process as a multi-task learning problem with anchor classification, box regression, and embedding learning. The JDE architecture chooses the Feature Pyramid Network (FPN) [22] as its base architecture in order to extract multi-scale features. In the detection branch, it exploits the anchor-based detection paradigm, which sets a certain number of anchors of different sizes in each position in the feature map. Meanwhile, in the re-identification branch, consistent with general re-identification, it regards the re-identification as a classification task for network design.

However, the object may appear in the middle of two anchors, resulting in an ambiguity between the detection task and re-identification task. Therefore, the anchor-based method will cause the bounding boxes and embedding features to fail to align. Some studies [23,24] have shown that the anchor-free method is more suitable for the detection branch of the one-shot framework. They directly classify the object and background from the feature map and regress the bounding box. Although the one-shot methods have the advantage in regard to time, their accuracy is usually lower than that of the two-stage methods [24]. They tend to overlook the conflicts caused by a joint learning of the different classification and re-identification tasks. Additionally, they do not take into account the importance of the motion features on which the tracking performance often hinges. Inspired by the other computer vision tasks, such as action recognition [25] and semantic segmentation [21], we have improved the performance of one-shot MOT tracking with the proposed MAC-MOT model.

3. Materials and Methods

3.1. Overview

In this section, we introduce the MAC-MOT in detail. The overall framework of the network is described in Figure 1. The MAC-MOT consists of five components, a backbone, a motion-enhanced attention module, a dual correlation attention module, a detection branch, and a ReID branch. MAC-MOT takes adjacent frames as input, denoted by $I_t$ and $I_{t-1}$. The backbone is exploited to extract multiple scale features from $I_t$ and $I_{t-1}$. Then, the MEA utilizes the temporal difference between the features from the two frames to enhance the motion information. We are able to obtain the features that contain more motion information, denoted by ${\phi }_t$. Next, the DCA module is proposed to decouple ${\phi }_t$ into a detection feature $F_{det}$ and a re-identification $F_{reid}$. Finally, the detection branch tasks the $F_{det}$ as input to get the bounding boxes of the objects, while the re-identification branch takes the $F_{reid}$ as input to get the embedding feature of each object.

3.2. Motion Enhance Attention

We designed a MEA module in the one-shot method to focus on the motion salient features, while suppressing the irrelevant information in the background. MEA uses the temporal difference operation of the adjacent frame level features to enhance the motion related features in a channel-wise manner. Figure 2 shows two feature maps, $F_t$ and $F_{t-1}$, with a tensor dimension of $C\ast H\ast W$, in which $C$ is the channel, $H$ is the height, and $W$ is the width. $F_t$ and $F_{t-1}$ are the output of the backbone features extracted from the previous section. We first use the global adaptive pooling (GAP) to aggregate the feature maps across the spatial dimensions. This gives us a result of $C\ast 1\ast 1$, which generates the channel-wise importance weights. This average pooling is along the $H$ and $W$ dimension. Then, these intermediated vectors go through two weight-sharing 2D convolutions with a kernel size of $\raisebox{1ex}{$C$}\!\left/ \!\raisebox{-1ex}{$r$}\right.\ast 1\ast 1$, which are exploited to convert the channel dimensions to $\raisebox{1ex}{$C$}\!\left/ \!\raisebox{-1ex}{$r$}\right.$. This is effectively a process of feature dimension conversion. This conversion process reduces the channel size to refine the import features. We use an aggregation parameter r to control the output sizes.

The pipeline of the difference calculation can be formulated as:

$$diff=Conv1\left(GAP(F_t)\right)-Conv1\left(GAP(F_{t-1})\right),$$

(1)

Subsequently, another 2D convolution with a kernel size of $C\ast 1\ast 1$ is applied on the diff operator, in order to recover the dimension of the diff as for the input $F_t$. The attention weight is obtained by:

$$w_t=\sigma \left(Conv2\left(diff\right)\right),$$

(2)

The operator $\sigma$ represents the sigmoid function, which is used to normalize the values of the weights to 0–1. We utilize channel-wise multiplication to enhance the motion-salient features ${\phi }_t$.

$${\phi }_t=\left(1+w_t\right)· F_t,$$

(3)

In summary, the MEA uses the channel differences as the weight, in order to obtain better temporal information and motion features.

3.3. Dual Correlation Attention

Multi-task learning often suffers from task incompatibility, which leads to a decline in performance and even task failure. We propose a dual correlation attention to learn the commonalities and specificities of features for the detection and ReID tasks. For specificities learning, we use the self-attention mechanism to obtain the feature representation for each task. For commonalities learning, two tasks can be learned by an elaborately designed correlation mechanism.

The structure of our dual correlation attention module is presented in Figure 3. The feature ${\phi }_t$, which is the output of the MEA module depicted in Figure 2, is split into two branches through two convolutions of the same structure with a kernel size of $C\ast 3\ast 3$. The outputs are denoted by ${\phi }_1$ and ${\phi }_2$. Following the convolution, a reshape operator is implemented. Specifically, ${\phi }_1$ and ${\phi }_2$ are then reshaped to $M_1$ and $M_2$. The reason for this reshaping operation is due to a dimension matching in doing the matrix multiplication computation. As a result, the spatial two-dimensional distribution of the feature map is transferred to a one-dimensional distribution, where $N=H^{\prime }\ast W^{\prime }$. Subsequently, we perform the matrix multiplication on $M_1$ or $M_2$ and its corresponding transpose. A row softmax function is exploited to obtain the self-attention weight maps $M_{T_1}$ and $M_{T_2}$ for each task. Again, we perform the matrix multiplication between $M_{T_1}$ and $M_{T_2}$ to learn the commonalities between the different tasks, and then a row softmax is followed to generate the correlation attention weight maps $M_c$. $M_c$ is added to $M_{T_1}$ and $M_{T_2}$, aiming to get the enhanced representation, denoted by $M_{S_1}$ and $M_{S_2}$. The enhanced representations are fused with ${\phi }_t$ by the attention to prevent information loss.

3.4. Loss Function

The one-shot methods are composed of the detection task and re-identification task. Thus, we need to design corresponding losses for the two tasks at the same time.

Detection Branch. For a bounding box $b^i$ = {$x_1^i, y_1^i, x_2^i, y_2^i$}, we need to know its center point coordinates, width and height. The detection branch contains two heads, a heatmap head and a box size head. The heatmap head is responsible for estimating the locations of the object’s center. The response at a location where the object is in the heatmap is expected to be one. We define {$C_x, C_y$} to be the center point of the object. Then, its location on the feature map is obtained by dividing the stride as follows:

$$(\widetilde{C_x}, \widetilde{C_y})=(\lfloor \frac{C_x}{stride}\rfloor \lfloor \frac{C_y}{stride}\rfloor ),$$

(4)

The stride represents the down sampling rate of the network. The value of our backbone network is 4. The heatmap response at the location $\left\{x,y\right\}$ is computed as follows:

$$M_{xy}={{\displaystyle \sum }}_{i=1}^{N}ex{p}^{\frac{-{\left(x-{\widetilde{c_x}}^i\right)}^2+{\left(y-{\widetilde{c_y}}^i\right)}^2}{2{\sigma }^2}},$$

(5)

where N represents the number of objects in the image and $\sigma$ represents the standard deviation. The heatmap loss is defined as follows:

$$L_{heatmap}=-\frac{1}{N}{\displaystyle \sum }_{xy}\{\begin{array}{c}{\left(1-\widehat{M_{xy}}\right)}^{\alpha }log\left(\widehat{M_{xy}}\right), if M_{xy}=1\\ {\left(1-M_{xy}\right)}^{\beta }{\left(\widehat{M_{xy}}\right)}^{\alpha }log\left(1-\widehat{M_{xy}}\right) otherwise\end{array},$$

(6)

$\widehat{M_{xy}}$ is the estimated heatmap, M_xy is the ground truth, and $\alpha$ and $\beta$ are the parameters. For each bounding box $b^i$ = {$x_1^i, y_1^i, x_2^i, y_2^i$}, we define the size of the bounding box as $s^i$ = {$x_2^i-x_1^i, y_2^i-y_1^i$}. The box size head output is denoted as ${\widehat{s}}^i$. The box loss is defined as follows:

$$L_{box}={{\displaystyle \sum }}_{i=1}\Vert s^i-{\widehat{s}}^i\Vert ,$$

(7)

Re-identification Branch. The goal of the identity embedding is to generate features that can distinguish different objects. We hope that the difference in the same object is much smaller than that of the different object. We apply a convolution layer to exact the identity embedding features for each location. We usually train a re-identification representation model as a classification task. All object instances of the same identity in the training set are treated as one class. The identity embedding loss is defined as follows:

$$L_{identity}=-{\displaystyle \sum }_{i=1}^N{\displaystyle \sum }_{k=1}^{K}L{\left(K\right)}^{i}log\left(p\left(K\right)\right),$$

(8)

where $p\left(K\right)$ represents the identity embedding features, $L{\left(K\right)}^i$ represents the distribution of the identity embedding features, and $K$ is the number of classes.

Total Loss. We train the entire network to optimize the parameters of the two branches together. We add two trainable parameters to balance the two tasks, denoted by $P_{det}$ and $P_{reid}$. The total loss is formulated as:

$$L_{total}=e^{-P_{det}}\ast \left(L_{heatmap}+L_{box}\right)+e^{-P_{reid}} \ast L_{identity}+\left(P_{det}+P_{reid}\right),$$

(9)

4. Experiments

In this section, we firstly introduce the relevant experimental settings. The proposed MAC-MOT has shown robust results on the MOT17 [26] and MOT16 [26] compared with the SOTA. Ablation study is also conducted on the MOT17 [26] testing dataset to verify the effectiveness of the motion enhance attention (MEA) and dual correlation attention (DCA). We also demonstrate the impact of the parameters on network performance. Visualization of the tracking results is also presented to illustrate the superior performance of the proposed approach.

4.1. Experimental Settings

The MOT16 [26] and MOT17 [26] datasets are used in our experiments. We first adopted the pre-trained Deep Layer Aggregation 34 (DLA-34) [23] model on the Common objects in Context (COCO) dataset [27] to initialize our backbone. The MOT17 training set is used to train this proposed end-to-end network. A total of 30 epochs are trained, with the Adam optimizer. The batch size is set to 12. The initial learning rate is 1 × 10⁻⁴, and we decay the learning rate to 1 × 10⁻⁵ at the 20th epoch. The hardware configuration is a 4-GPU machine, with RTX 2080Ti GPUs.

Hyperparameter r, used in the MEA model as depicted in Figure 2, controls the degree of the aggregation of features. We use two sizes of convolution kernels $64\ast 1\ast 1$ and $\raisebox{1ex}{$64$}\!\left/ \!\raisebox{-1ex}{$\mathrm{r}$}\right.\ast 1\ast 1$ in our experiments. We have determined the value of r through experiments. The hyperparameter in the DCA module controls the kernel size of the convolution operator; we chose the convolution with a kernel size of $64\ast 3\ast 3$ for this experiment.

For the evaluation metrics, we follow the MOT challenge [28] to evaluate the MOT performance, including multiple object tracking accuracy (MOTA), multiple object tracking precision (MOTP), Identity F1 score (IDF1), the total number of ID switches (IDs), the MT (number of mostly tracked), and the ML (number of mostly lost). We evaluate our network on the MOT16 and MOT17 test sets and do not use additional training data.

4.2. Comparison Experiments

We compare our MAC-MOT with the SOTA approaches on the MOT16 and MOT17.

Table 1. Tracking result on MOT16. Red color represents best performance. Blue color represents second place.

Method	MOTA↑	IDF1↑	MT↑	ML↓	IDs↓
DeepSort-2 [29] ICIP 2017	61.4	62.2	32.8	18.2	781
RAW16wVGG [30] CACV 2018	63.0	63.8	39.9	22.1	482
TubeTK [31], CVPR 2020	64.0	59.4	33.5	19.4	1117
JDE [20] ECCV 2020	64.4	55.8	35.4	20.0	1544
HOGM [32] ICPR 2018	64.8	73.5	40.6	22.0	1544
CNNMTT [33] CMTA 2019	65.2	62.2	32.4	21.3	946
POI [34] ECCV 2016	66.1	65.1	34.0	21.3	805
CTrackerV [35] ECCV 2020	67.6	57.2	32.9	23.1	1897
FairMOT [24]	69.3	72.3	40.3	16.7	815
MAC-MOT(Ours)	71.7	70.7	39.3	18.3	1393

and

Table 2. Tracking result on MOT17. Red color represents best performance. Blue color represents second place.

Method	MOTA↑	IDF1↑	MT↑	ML↓	IDs↓
TubeTK [31] CVPR 2020	63.0	58.6	31.2	19.9	4137
CTracker [35] ECCV 2020	66.6	57.4	32.2	24.2	5529
CenterTrack [23] ECCV 2020	67.8	64.7	34.6	24.6	3039
DeepSort [29] ICIP 2017	60.3	61.2	31.5	20.3	2442
FairMOT [24]	69.8	69	39.4	21.8	3960
MAC-MOT(Ours)	70.1	69.8	38.2	20.0	4392

show the results on the MOT16 and MOT17 benchmarks, respectively. The best performance is highlighted in a red color, while second place is highlighted in a blue color. Table 1 shows the comparisons on the MOT16 benchmark. MAC-MOT improves at least 2.4 points on the MOTA. Table 2 shows the performance of the MAC-MOT on the MOT17; our approach has a minimum of a 0.3 points improvement on the MOTA and 0.8 gains on the IDF1. Regarding the MT and ML in the two tables, we still report these performances to provide the entire picture of the proposed method, as well as other SOTAs.

There are two reasons that the proposed model has a better performance. On the one hand, the MEA enhances the moving feature of the object, which makes it more distinguishable. On the other hand, the DCA module reduces the conflict between the detection and re-identification during training. At the same time, we also notice that our approach leads to an increase in IDs in both the MOT16 and MOT17. The reason for this degraded performance is that adding two modules causes the stability of the one-shot framework to decrease.

4.3. Ablation Studies

In order to verify the role of each module, we studied each component in our tracking framework. All experiments are evaluated on the MOT17 testing set. The results of the experiments are shown in

Table 3. Experiment results of the ablation studies.

Method	MOTA↑	MOTP↑	IDF1↑	MT↑	ML↓	IDs↓
baseline	69.8	80.3	69	39.4	21.8	3960
MEA	69.9	80.3	69.6	40.3	19.5	4665
DCA	70.4	80.1	70.1	40.3	20.4	4416
MEA + DCA	70.1	80.3	69.8	38.2	20.0	4392

. We regard the network without the MEA and DCA as the baseline. The impact of the MEA and DCA is evaluated and discussed both in their individual and combined form. As can be seen from the Table 3, the baseline result has a 69.8 MOTA and 69 IDF1. By adding only the MEA module, it achieves a 0.1 point improvement on the MOTA and 0.6 gains on the IDF1. This demonstrates an improvement contributed by adding the attention on using the motion information. Adding the DCA module achieves a 0.6 point gain on the MOTA and a 1.1 gain on the IDF1. This result showcases the performance gain using the self-attention and correlation attention. When combining the MEA and DCA simultaneously, the result achieves a 0.3 point improvement on the MOTA and a 0.8 gain on IDF1. This combination approach does not combine the individual gain. This could be due to a combined attention effect which could cancel the other’s attention mechanism.

4.4. Parameter Analysis

There is a feature dimension conversion coefficient r in the MEA module, which reflects the degree of the feature aggregation. In order to find the suitable parameters, we conducted experiments on the MOT17 verification set. As shown in Figure 4, we chose four values of r: 4, 8, 16, and 32. We find that the MOTA is the highest when r is 8, and the IDF1 continues to decrease as r increases. Here, we conclude that with the current experiment settings, the hyperparameter r equaling to 8 provides the best performance.

4.5. Track Visualization Results

The MAC-MOT has a better tracking ability than the SOTA one-shot method [24] in a real scene sequence. As shown in Figure 5, when the object is far away from the camera and the object is blurry, the simple one-shot method will fail to track the object; the object is missed. Our approach has improved this situation due to the enhanced motion-related feature.

5. Conclusions

In this paper, we propose a novel network named MAC-MOT, which has a moduled attention in two parts: a motion enhance attention module (MEA) and a dual correlation attention module (DCA). The proposed MAC-MOT improves on previous one-shot approaches. The MEA plays a role in capturing the motion information which is ignored by the previous approaches. When an object is far away from the camera or the object is blurred due to motion, our approach can still keep track of the target objects while the previous counterpart fails. Meanwhile, the previous approach has a separate detection and re-identification task that produces ambiguity when the network parameters are optimized in a unified framework. The MAC-MOT uses the dual correlation module to obtain features that are more suitable for different tasks simultaneously. This strategy reduces the competition conflict caused by multi-task joint learning during training. In MOT challenges, our approach has achieved the competitive performance on the MOT16 and MOT17 datasets. Additionally, we also find that the simple combination of the two modules does not achieve the superposition of benefits. In the future, we will further investigate and improve the structure of the network to achieve a better tracking performance.