Multi-Object Tracking Based on Re-Identification Enhancement and Associated Correction

Abstract

During the Multi-Object Tracking (MOT) process, the effectiveness of Re-Identification (Re-ID) is typically contingent upon the performance of detection. In particular, detection errors may result in the blurring of Re-ID features, thereby diminishing the efficacy of Re-ID. In particular, Re-ID exhibits poor robustness in intricate scenarios such as distance and scale alterations, occlusion, and crowding. Meanwhile, the frequent occurrence of object occlusion can result in the interruption of object tracking, thereby impacting the tracking efficiency. To address these issues, a novel MOT method approach that leverages Re-ID enhancement and association correction is proposed. Through an unanchored, one-stage backbone network with CenterNet as the detector, the object position is detected by predicting the heatmap of the object center point. A constraint on the Predicted Distance Difference Loss (PDDL) is then designed for the response value of the predicted heatmap, effectively enhancing the detection accuracy and reduce the detection bias. To address the issue of Re-ID reliance on detection accuracy and enhance its robustness, an Effective Feature Graph Convolutional Neural network (EFGCN) is proposed. This approach expands learnable effective features by constructing a graph model that enhances them through a GCN model of heatmap true value locations and surrounding response value locations. Additionally, a target trajectory Association Correction Module (ACM) is introduced in the association phase to handle interruptions and trajectory fragments caused by occlusion. By improving data association for tracking judgment when dealing with occluded targets and considering the relationship between trajectory fragmentation and the detection of low-scoring targets, attempts are made to continue with tracking occluded targets to form complete object associations. The proposed method is evaluated on the MOT16 and MOT17 datasets and compared with other algorithms. The results demonstrate that it is highly effective.

1. Introduction

Multiple-object tracking (MOT), as a high-level task in computer vision, has been widely applied in various fields such as video surveillance [1,2,3], human behavior recognition, and autonomous driving [4,5,6]. One of the advantages of video-based MOT [7,8,9,10,11,12,13,14] is its ability to accurately localize targets continuously in videos while maintaining their identity information, unchanged, when the appearance or the surrounding environment changes. This allows for the output of complete motion trajectories for tracked objects. As a result, it has gained increasing attention from researchers.

The MOT task can be classified into two categories based on the use of video sequences: online and offline tracking. While offline tracking is capable of achieving better tracking performance by utilizing global information from videos, online tracking is more suitable for real-world applications. However, due to the fact that online tracking is unable to take advantage of subsequent image information in videos, its tracking accuracy relies more heavily on the detection precision of current image targets. Additionally, there has been a growing trend towards targeting specific types of objects within scenes, as many videos contain large numbers of pedestrians. Consequently, pedestrian tracking algorithms currently account for the majority of multi-object tracking methods [15,16,17,18,19,20]. Nevertheless, there are still challenges in improving both real-time performance and tracking robustness simultaneously.

As object detection algorithms continue to improve, Tracking by Detection (TBD) [21] has become a popular tracking strategy. TBD utilizes detectors to distinguish between foreground and background objects in MOT tasks, improving tracking robustness through the tracker continuous communication with the detector while predicting target positions. However, in complex scenarios such as scale transformations and frequent occlusions, target appearance features can be contaminated, leading to poor tracking results. To address this issue, researchers have utilized Re-ID deep learning models for feature extraction, followed by similarity analysis using Euclidean Distance, Cosine Dsitance, and other metric functions to optimize accuracy and differentiation [22].

The methods of Re-ID [23] feature extraction are different; the (Joint Detecting and Embedding, JDE) algorithm proposed by Wang et al. [24] merges the pedestrian re-recognition model into the detector network, and outputs the detection information of the image and the Re-ID feature map in parallel. The detection information is then used to obtain the corresponding feature vector of the target, which avoids the redundant computation of repeated feature extraction. However, the JDE algorithm has a poor detector effect in the case of pedestrians occluding each other, and pedestrian re-recognition relies too heavily on the results of the detector. Zhang et al. [25] proposed FairMOT, an excellent algorithm based on the JDE integration framework. The contradiction between detection and apparent features is alleviated by fusing deep and shallow features. CenterNet [5], based on anchorless detection, is used as a detector to reduce the ambiguity of Re-ID features. The tracking accuracy is further improved and real-time requirements are met.

CenterNet is a centroid-based detection algorithm with detection bias. During tracking training, classification is learned for the feature vectors of the Re-ID feature map at the center point of the target. The inability to accurately locate the effective feature positions on the feature map causes the Re-ID task to be overly dependent on the detection accuracy. Obtaining better tracking results not only improves the detection accuracy but also improves the performance of the association task. Background interference, illumination variations, pedestrian density, and target occlusion have always been difficult problems in multi-target tracking methods. Existing methods often treat short-term occlusion of a target by other objects as target disappearance. When the target reappears, a trajectory is reinitialized, which destroys the integrity of the target trajectory.

To address the above problems this paper proposes a multi-objective tracking algorithm (REACTrack) based on Re-ID enhancement and association correction. The main contributions of this paper are as follows:

• In CenterNet network, a predicted distance difference loss (PDDL) is designed for a center point detection heat map—suppress the response values of non-true Ground Truth (GT) locations in the predicted heat map. Make the high response value closer to the target truth position to improve the detection effect;
• Build an expanded learnable Effective Features GCN (EFGCN). Build a graph model for the relationship between the target GT position and the position around the GT. Ensure Re-ID feature quality and robustness;
• Design of Trajectory-Associated Correction Module (ACM). Under the influence of complex background and frequent occlusion of pedestrian targets, the target trajectory is corrected to improve the trajectory integrity.

The rest of the paper is organized as follows: Section 2 provides the related work on MOT. Section 3 explains REACTrack with detailed description for each stage. Section 4 gives the ablation studies and baseline comparison results and visualization. Finally, Section 5 concludes the paper by providing perspectives for future work.

2. Related Work

MOT can be categorized into two common paradigms: Tracking by Detection (TBD) and Joint Detection and Tracking (JDT). TBD is divided into two stages: in the first phase, an external detector is utilized to discover and localize all the targets of interest in each image frame. The detection response is represented as a rectangular bounding box. In the second phase, data association is performed on the detection responses to achieve tracking of multiple targets. The execution time of the TBD tracking system is approximately the sum of the execution time of each of the two phases. And it becomes larger as the number of targets in the scene increases, which poses a great challenge in building a real-time, multi-target tracking system. In order to reduce computation, researchers have recently proposed the JDT paradigm to perform target detection and target tracking simultaneously. Bergmann et al. [26] first proposed a transition paradigm from the detection-based tracking paradigm to the joint detection and tracking paradigm: the regression-based tracking paradigm, Tracktor. Tracktor transforms a target detector into a tracker by utilizing the bounding box regression of the target detector Faster RCNN [16] to directly predict the position of the target in the next frame. Compared to the detection-based tracking paradigm, Tracktor achieves better tracking performance.

In MOT, Re-ID is often utilized to introduce differentiation between targets to improve the matching accuracy of the tracking algorithm and reduce the false matches. Meanwhile, Re-ID’s target-specific re-recognition capability can help the trajectories of lost targets to re-match the reappearing targets and improve the tracking robustness. Bayraktar et al. [27] proposed an embedding generation module which is tightly coupled with the object detection module and a triplet-based matching module to achieve the fast Re-ID task of objects. Suljagic et al. [28] proposed an interesting similarity-based person Re-ID framework by adopting a Siamese neural network via shared weights. Due to its powerful modeling capability, GCN is widely used in the field of pedestrian re-identification. Yan et al. [29] and Shen et al. [30] constructed graph models at the image level, i.e., each node of the graph represents an image in order to take into account the relationships between images; however, they ignored the relationships between different body parts of pedestrians. Wu et al. [31] introduced a graph neural network to achieve contextual interaction between relevant regional features by utilizing pose-aligned connections and feature affinity connections. However, pose information extraction requires additional computation and is not integrated into the whole network for end-to-end training, which may lead to sub-optimal results. Shen et al. [32] collected candidate items from the output of detection and tracking via a deep neural network and adopted a classifier with codec architecture. They also introduced Re-ID feature information for data association. In addition, the commonly used MOT data association methods [24,25,33,34] only associate the detection frames with scores above a certain threshold with the trajectory, while ignoring the case when the score is below the threshold when the pedestrian is occluded, which leads to non-negligible real missed targets and fragmented trajectories. This is insufficient to depict the pedestrian’s motion state and can easily cause interruption of the trajectory and ID. Zhang et al. [7] proposed to perform tracking by associating each detection frame instead of only the high-scoring detection frames. For low-score detection frames, we utilize their similarity to small trajectories to recover real objects and filter out background detection.

In this paper, PDDL is proposed for centroid detection to make the predicted position closer to the true position of the target, which, in turn, improves the detection accuracy; EFGCN is constructed to improve the quality of Re-ID feature vectors since centroid detection has detection bias, which reduces the Re-ID effect; the detection information is used to set the tracking thresholds, select the available targets, carry out the correlation corrections to the existing trajectory segments, add an ACM to minimize the trajectory interruptions occurring due to occlusion, and optimize the correlation effect.

3. Methods

In this paper, a MOT algorithm, REACTrack, based on re-identification enhancement and association correction is proposed. The algorithm extracts features through DLA-34 network and uses CenterNet, which is based on anchorless detection as a detector. The differential distance loss PDDL for center detection is proposed to enhance the detection effect. Then, GCN is used to expand the effective feature range of Re-ID and enhance the quality of Re-ID features. Finally, we propose a target association correction method, ACM, to ensure the integrity of the target trajectory.

3.1. Overview

The overall network framework of REACMOT proposed in this paper is shown in Figure 1. First, the input images are fed into the backbone network (DLA-34) [25] with a uniform input image size of 1088 × 608 to extract the high-resolution feature maps (stride = 4), and the output feature maps with a size of 64 × 272 × 152. Then, the feature maps are input to CenterNet. PDDL is proposed in the detection branch to improve the detection effect. EFGCN is used in the Re-ID branch to expand the effective feature range in the center of the predicted object. Finally, the target association correction module is connected to reduce the trajectory interruptions occurring due to occlusion and obtain the complete trajectory.

3.2. Object Detection

The feature map extracted by the backbone network is fed to the CenterNet detector, which contains three parallel branches used to estimate the heatmap, center offset and bounding box size of the object. To improve the detection effect, we designed PDDL in heatmap to make the predicted position close to the GT position.

3.2.1. Heatmap Prediction Distance Difference

For the input image $I\in R^{W\times H\times K}$, where $K=1$, a center point heatmap is generated at the time of prediction, denoted as $\widehat{M}\in {\left[0,1\right]}^{\frac{W}{4}\times \frac{H}{4}\times K}$; 4 denotes 4-fold downsampling. When ${\widehat{M}}_{x,y,k}=1$, this point is the detection center point, otherwise it belongs to the background.

For (ground truth) GT values, the center point is denoted as $c=\left(c_x,c_y\right)\in R^2$. The center point is computed as $c=\left(\frac{x_1+x_2}{2},\frac{y_1+y_2}{2}\right)$, and its downsampled corresponding center point is $\widehat{c}=\lfloor \frac{c}{4}\rfloor$.

The image is labeled with $M\in {\left[0,1\right]}^{\frac{W}{4}\times \frac{H}{4}\times K}$ and the center point is noted as the GT point in the downsampled [128,128] image will be represented in the form of $M\in {\left[0,1\right]}^{\frac{W}{4}\times \frac{H}{4}\times K}$. The heat map response of GT satisfies a Gaussian distribution:

$$M_{x,y,k}=\exp \left(-\frac{{\left(x-{\widehat{c}}_x\right)}^2+{\left(y-{\widehat{c}}_y\right)}^2}{2{\sigma }_c^2}\right),$$

(1)

${\sigma }_c$ is the standard deviation associated with the target size.

The generated Gaussian kernel (target frame size) is placed on the corresponding position of the original image (image size), each point $M\in {\left[0,1\right]}^{\frac{W}{4}\times \frac{H}{4}\times K}$ ranges from 0–1 and 1 represents the center point of this target.

The Heatmap output from the Heatmap head is used to predict the target centroid position. The sizes of the Heatmap are 1 × H × W. If a position of the Heatmap has a value of 1, indicating the coincidence with the target centroid, it is expressed as the GT response value. However, predicting the target center point via heatmap is prone to the situation that the predicted position is shifted and cannot hit the target GT position accurately, as shown in Figure 2. The size of the heat map is 272 × 152. Numbers are response values and axes are labeled as coordinate positions in the heatmap. (a) is the true value GT heatmap of the target, the star marking is the target GT position (GT response value is 1); (b) is the predictive heatmap and the circle marking is the position of the highest response value of the predicted target, i.e., the predictive heatmap target position. It can be seen that there is a certain deviation between the predicted position and the GT position. Although this small deviation has a small impact on the detection accuracy, since the Re-ID feature map only sets the learnable feature vectors at the GT position, the small deviation can also lead to the inability to determine the learnable feature position, which affects the Re-ID effect.

To solve the problem, this paper defines the distance from the predicted position to the corresponding true position as the distance difference, and constructs the predicted distance difference loss with the average distance difference of high response values to suppress the magnitude of the predicted response values at non-true positions, so that the predicted position is closer to the true value and further improves the detection accuracy. With the object’s true value position as the center, the farther the predicted position with higher response value is from the true value, the greater the loss brought about and the stronger the suppression effect on its response value; on the contrary, the closer the true value position is, the smaller the suppression effect on the high response value, thus bringing the predicted high response value closer to the true value position and realizing the correction of the prediction distance difference. For example, the predicted response value of the true location in the predicted heat map in Figure 2 is 0.93, while there are similar as well as higher predicted response values at the surrounding non-true locations, such as the predicted location being 0.94. After constructing the predicted distance difference loss, the predicted distance difference at the true location is zero, which has no suppression effect, but the high response values at the remaining locations are suppressed to varying degrees, thus ensuring the highest response value at the true location and correcting the predicted location to the GT value.

3.2.2. Heatmap Loss

Heatmap loss includes response value prediction loss and prediction distance difference loss, prediction loss is defined as the error between the predicted response value and the true response value, due to the problem of imbalance in the ratio of positive and negative samples and difficult samples for centroid detection, in order to reduce the effect of sample inhomogeneity, the loss function follows the Focal Loss [35] function in CenterNet, which is expressed as follows:

$$L_p=-\frac{1}{N}{\displaystyle \sum _{xy}\{\begin{array}{c}{\left(1-{\widehat{M}}_{xy}\right)}^{\alpha }\ln \left({\widehat{M}}_{xy}\right),\hspace{1em}{M}_{xy}=1\\ {\left(1-M_{xy}\right)}^{\beta }{\widehat{M}}_{xy}{}^{\alpha }\ln \left(1-{\widehat{M}}_{xy}\right),other\end{array}},$$

(2)

where N is the number of objects, ${\widehat{M}}_{xy}$ is the predicted response value of the heatmap at (x, y), $M_{xy}$ is the GT response value, and the preset parameters $\alpha ,\beta$ for Focal Loss are $\alpha =2,\beta =4$.

The detection deviation loss is defined as the deviation of the distance from the predicted position to the corresponding GT position. Considering that a heatmap detects the object location by predicting the response value, it is not necessary to guarantee the correspondence between the predicted location and the GT object, so the principle of proximity is used to select the corresponding GT object for each predicted location to calculate the loss.

For each high response position $\left({\widehat{c}}_x^i,{\widehat{c}}_y^i\right)$ of the predicted heatmap, the nearest GT position (the coordinate position with a response value of 1) is selected and the distance between the two coordinates is calculated as the loss. Here, the distance is the 2D coordinate point spacing, so L₂ distance is chosen as the loss. A response threshold of λ = 0.5 is set to filter high response values to ensure effective constraints on detection deviations, and the loss calculation is defined as follows:

$$L_d=\frac{1}{M}{\displaystyle \sum _{i=1}^M\min \left({\Vert \left({\widehat{c}}_x^i,{\widehat{c}}_y^i\right)-C\Vert }_2\right)},$$

(3)

where M is the number of predicted response values greater than λ, $\left({\widehat{c}}_x^i,{\widehat{c}}_y^i\right)$ is the predicted object location, $C=\left\{\left(c_x^j,c_y^j\right),j\in \left[1,N\right]\right\}$ is the set of GT position coordinates.

At the same time, considering that the detection deviation loss is a constraint on the deviation of high response value positions around the object centroid, it is a constraint adjustment based on the better response value prediction results. Therefore, the training of the heatmap should focus on response value prediction, and the detection deviation constraint should be performed when the heatmap response value prediction is better. However, the prediction heatmap is poor at the early training stage, and the detection deviation loss is a distance loss, which is larger than the response value prediction loss, so the constraint at this time will cause the $L_p$ fail to converge quickly and affect the training effect of response value prediction. In the later stage of training, when the response value prediction achieves better results, a stronger detection deviation constraint is needed to correct the predicted positions. Therefore, similar to multi-task training, the losses of the heatmap need to be adjusted to meet the training requirements instead of being directly summed. In order to adjust the loss, this paper adds a linear enhancement factor $e^{-L_P}$ to the deviation loss and adjusts the detection deviation loss by the change in the value of the response value prediction loss $L_p$ convergence process to ensure that the constraint is gradually increased when a better detection effect is achieved. The parameter, μ, is then set to adjust the deviation loss to be of the same order of magnitude as the predicted loss, which can be chosen according to the value of the loss, and μ = 0.1 is taken in the experiment.

The final heatmap loss is defined as follows:

$$L_{hm}=L_p+e^{-L_p}\cdot \mu L_d,$$

(4)

The above equation shows that when $L_p$ is large, the coefficient $e^{-L_P}$ is small, so $L_d$ has almost no effect on $L_{hm}$. However, as $L_p$ decreases, the detection deviation loss gradually increases the constraint strength, making the constraint process smoother and obtaining better results.

3.2.3. Offset and Box Loss

Box head outputs a feature map of size W × H × 4, which is used to estimate the distance from the center of the target to the four edges of the detection box. Offset head outputs a feature map of size W × H × 2, which is used to estimate the offset compensation of the center of the target. The offset compensation here is the loss of accuracy caused by the rounding operation when calculating the GT heatmap. Let a GT detection frame in the image be $\left(c_x^i,c_y^i,w_i,h_i\right)$. The distance from the center point to the detection frame is $d_i=\left(w_i/2,h_i/2,w_i/2,h_i/2\right)$. Since the sampling multiplier under the heatmap is 4, the compensation for the center point offset is $b_i=\left(c_x^i/4,c_y^i/4\right)-\left(\lfloor c_x^i/4\rfloor ,\lfloor c_y^i/4\rfloor \right)$. ${\widehat{d}}_i$ and ${\widehat{b}}_i$ be the predicted values and the loss function follows the L₁ distance in CenterNet, defined as follows:

$$L_{box}={\displaystyle \sum _{\mathrm{i}=1}^I{\Vert b_i-{\widehat{b}}_i\Vert }_1}+{\Vert d_i-{\widehat{d}}_i\Vert }_1.$$

(5)

3.3. Re-ID Enhance

Pedestrian re-identification is the process of extracting recognizable features from the pedestrians in the cropped frame so that the same pedestrians can be accurately identified when they reappear.

The Re-ID head feature map size is 128 × 272 × 152, and each feature point contains a 128-dimensional feature vector. An Effective Feature Graph Convolutional Network EFGCN is proposed. Expanding the range of learnable effective features, enhancing the Re-ID effect, and improving the quality of ID embedding feature vectors.

Currently, only the feature vector at the target GT is used as the Re-ID effective feature for training, so that the detection accuracy will have an impact on Re-ID. Moreover, and the Re-ID will also depend on the detection accuracy, and if no valid features are hit, it will directly affect the effectiveness of tracking. The purpose of the Re-ID effective feature setting process is to determine the location via the object GT heat map and then find the corresponding location on the Re-ID feature map to determine the effective feature vector. However, there is a detection bias in centroid detection, and when the highest response of the marker does not fall in the GT position; even if the deviation of the localization information is very small, it will reduce the feature quality of Re-ID and affect the Re-ID effect.

Expanding the effective feature range of Re-ID can alleviate the bias problem of detection and the chance of hitting a feature can be improved. The relationship between the heatmap GT location and the location around the GT is modeled using GCN.

Construct the graph $G\left(\nu ,e\right)$,with N nodes and where the feature dimension of each node is 128. $v$ denotes the set of each node $x$ of the graph structure, and $e$ is the set of edges formed by the vertices in the set $v$ with the following expression:

$$v=\left\{x_g,x_{l1},x_{l2},\dots ,x_{l8}\right\},$$

(6)

$$e_{gl}=\left(x_g,x_l\right)\in e,$$

(7)

inspired by [35], the relationship between the nodes is represented by (7), i.e., the effective feature expansion range threshold is defined as 0.5, forming an adjacency matrix $\widehat{K}$.

$$\widehat{K}={\left\{{\displaystyle {\sum }_l\left[K_{\left(g,l\right)}+I_n\right]}\right\}}^{-1}\left[K_{\left(g,l\right)}+I_n\right],$$

(8)

$$K_{\left(g,l\right)}=\frac{e^2\left(x_g,x_l\right)}{{\displaystyle {\sum }_{l=1}^N{e}^2\left(x_g,x_l\right)}}.$$

(9)

K is a key component of the GCN, and each element in K should be non-negative and have coefficients between 0 and 1, so K is normalized, where $I_n\in {\mathbb{Z}}^{N\times N}$ is the unit matrix. To accurately recognize the same target in a continuous video image, the feature map, $F\in R^{128\times 272\times 152}$, is extracted via EFGCN, and the target is distinguished by the similarity of the feature vector, $F_{x,y}\in R^{128}$, at the target GT.

Finally, after EFGCN, as shown in Figure 3, the feature vectors of all nodes with response values greater than the threshold and the GT position node features are aggregated within the threshold that satisfies the condition, which represents the expanded effective features and corresponds to the Re-ID feature map. After the expansion of the range of effective features, the tolerance of the re-id to detection bias can be improved to ensure the re-id features.

We learn Re-ID features through a classification task, where pedestrians of the same identity are considered to be of the same category. The target feature vector $F_{x,y}$ at GT is passed through a linear classification layer to obtain its probability value $P=\left\{p\left(k\right),k\in \left[1,K\right]\right\},$. For classifying each ID, where K is the number of identity categories, i.e., the total number of IDs. Re-ID loss is calculated as follows:

$$L_{id}=-{\displaystyle \sum _{i=1}^N{\displaystyle \sum _{k=i}^K{Y}^i}}\left(k\right)\ln \left(p\left(k\right)\right),$$

(10)

where, $Y^i\left(k\right)$ denotes the true ID probability distribution of the i-th object.

3.4. Multi-Task Training

To train both the detection task and the Re-ID task, the two tasks are dynamically balanced using uncertainty loss [36], which is calculated as follows:

$$\begin{array}{l}{L}_{de}=L_{hm}+L_{box},\\ L=\frac{1}{2}\left(\frac{1}{e^{w_1}}{L}_{de}+\frac{1}{e^{w_2}}{L}_{idG}+w_1+w_2\right),\end{array}$$

(11)

where $w_1$ and $w_2$ are learnable parameters for balancing tasks with initial values of −1.85 and −1.05, respectively.

3.5. Association Correction Network

In the MOT process, the target trajectory is often interrupted due to problems such as background interference and inter-target occlusion, and the target association using the Hungarian algorithm [37] may track the target, but the effect has limitations. For example, a pedestrian from the beginning of the occlusion to the end of the occlusion process, before and after the occlusion of the tracking target ID has been transformed, that is, the tracking trajectory has occurred to interrupt the situation. To address this phenomenon, this paper improves the Hungarian algorithm based on BYTE [7] and proposes a target association correction module to reduce the trajectory interruptions occurring due to occlusion, i.e., tracking the phenomenon of target ID transformations and obtaining a complete trajectory, as shown in Figure 4.

First, the detected pedestrian detection boxes are inputted into the association network, and the thresholds, $T_h$, $T_l$, and $T_{th}$, are set to denote the detection score high score threshold, low score threshold, and tracking threshold for data association matching, which are categorized into the high score detection boxes ($I_h$) and the low score detection boxes ($I_1$) through the confidence threshold.

Secondly, $I_h$ is IoU (Intersection-Over-Union) matched with the result of Kalman filter prediction of the detection box of the previous frame to calculate the similarity between the detection box and the prediction box; the extracted Re-ID appearance features are fused with the result of IoU matching for feature fusion, and the matching optimization is carried out by using the improved Hungarian algorithm, which is named as the Weighted Hungarian Algorithm, and the weighted cost function is:

$$C=\alpha C_{IoU}+\beta C_{\mathrm{Re}-ID},$$

(12)

α, β are normalized weighting factors.

The following three types of matching results are obtained: the successfully matched trajectory ($T_1$), the unmatched high score detection box ($I_{hremain}$), and the unmatched trajectory ($T_{remain}$). For $T_1$, the trajectory is updated using the Kalman filter, and for $I_{remain}$, it is initialized as a new trajectory to participate in the next association process, which completes the first association between the detection box and the prediction box. The trajectory ($T_{remain}$) is associated with the low-scoring detection box ($I_1$) for the second time, and in order to ensure the integrity of the trajectory and improve the robustness, the association correction module ACM is embedded.

Specific description of ACM: the target ID during the tracking process is saved and the average value of the height-width of the tracking box of the tracked target is calculated as $\overline{H}$ and $\overline{W}$. Set the border height of target i of the current t-frame to $B_i^h$ and width to $B_i^w$, perform the comparison, and set the threshold value $\in \left\{0.99,1.01\right\}$. Let the center of target i in frame t − 1 be $P_i^{t-1}\left(x_i^{t-1},y_i^{t-1}\right)$; let the center of object i in frame t be $P_i^t\left(x_i^t,y_i^t\right)$. Comparisons are made using the Euclidean distance θ, θ $\in$ {0, 2}; If $B_i^h$ and $B_i^w$ are $\in \left\{0.99,1.01\right\}$, and ${\sqrt{{\left(x_i^{t-1}-x_i^t\right)}^2+{\left(y_i^{t-1}-y_i^t\right)}^2}}^{}\le \theta$, then the front and back frames are the same target.

After the second association, three results are obtained: for the successfully matched trajectories ($T_2$), they are fed into the Kalman filter again to update the trajectory information. For unmatched low-score detection boxes ($I_{lremain}$), they are treated as background deletion. For still-unmatched trajectories ($T_{re-remain}$), if they do not match $I_1$ for 30 consecutive frame rates, they are deleted. Instead, it is added to the tracking sequence. Finally, the obtained tracking sequence is predicted by Kalman filtering, and then associated with $I_h$ and $I_1$ in turn, and the above steps are repeated.

4. Experiments

The experiments were conducted on a 64-bit Ubuntu 20.04 system with GPU 3070Ti *3 and Cuda version 10.2. The open source framework used was PyTorch 1.3.1 and Python version 3.8. We evaluated the datasets, MOT16 and MOT17, from the MOT Challenge benchmark and demonstrated the effectiveness of our approach on the multi-objective tracking problem and compared our model with other recent methods.

The MOT16 dataset [38] has 14 video sequences, including 11,235 video frames, 1342 identity IDs, and 292,733 object frames. The MOT17 dataset [38] contains a total of 14 videos, divided into 7 training datasets and 7 test datasets. MOT17 has high annotation accuracy and covers a variety of multi-object tracking scenarios, covering scenarios where various challenges may be encountered during multi-object tracking, and the tracking algorithm needs good robustness and generalization to achieve good results with the dataset.

4.1. Evaluation Indicators

In order to comprehensively evaluate the tracking performance of the algorithm, several metrics [39,40,41,42] are usually combined to evaluate the tracking performance of the model. The IDs and IDFs are important indicators of the tracking robustness of the model, such as the accuracy and recall of the object IDs, the FP (False Positives) number of hypothetical bounding boxes in the whole sequence that cannot match the bounding boxes of the real track, and the FN (False Negatives) number of bounding boxes in the whole sequence that cannot match the hypothetical bounding boxes of the real track. The MOTA (Multiple Object Tracking Accuracy) is one of the most important evaluation metrics, as is the HOTA (Higher Order Tracking Accuracy). The MT (Mostly Tracked) indicates the percentage of objects that are tracked successfully for more than 80% of the trajectories; the ML (Mostly Lost) indicates the percentage of objects that are tracked successfully for more than 80% of the trajectories. ↑ indicates that larger values are better, ↓ indicates that the smaller the value, the better.

4.2. Ablation Experiment

In this section, we evaluate our work through ablation experiments and comparisons. The experiments use CenterNet object detection model parameters on the COCO [43] dataset to initialize the algorithm model using a deep aggregation network (DLA-34) as the backbone network, which uses a tree structure to fuse semantic information and spatial features from the shallow and deep layers. The Adam optimizer is used to train 35 epochs on the MOT16 dataset, with the batch size set to 12, the initial learning rate set to $e^{-4}$, and the attenuation at the 25th epoch set to $e^{-5}$. The response threshold for predicting the distance difference loss is set to 0.5; the Embedding matching threshold is 0.4; the IoU matching threshold is 0.5; the high detection score threshold is 0.6; the low detection score threshold is 0.2; and the trajectory initialization threshold is 0.7. In order to verify the effectiveness of each module, the corresponding ablation experiments were conducted on the MOT16 dataset, and common MOT official evaluation methods were used to verify the effectiveness of each part on the key parameters, and the specific results are shown in

Table 1. REACTrack ablation experiments. √ denotes addition of this module to the base line, which is FairMOT. PDDL, EFGCN, and ACM denote the Predictive Distance Difference Loss, Effective Feature Graph Convolutional Neural Network, and Association Correction modules, respectively. The best results are shown in bold.

Baseline	PDDL	EFGCN	ACM	MOTA ↑	IDF1 ↑	MT ↑	ML ↓	FP ↓	FN ↓	IDS ↓
√				71.7	75.4	33.2	24	923	7934	947
√	√			73.9	78.2	35.9	20.3	847	7713	495
√	√	√		77.2	80.3	37.1	17.8	756	7535	457
√	√	√	√	81.1	82.4	49.5	14.2	672	6952	292

.

By adding the Predicted Distance Difference Loss PDDL, MOTA improves by 2.2% and IDF1 improves by 2.8%, and the effective enhancement of IDF1 proves that the quality of trajectory tracking is optimized. By adding PDDL + EFGCN, MOTA is improved from 73.9 to 77.2, and IDF1 is improved from 78.2 to 80.3, which shows the effect of the two modules working together to effectively improve the detection accuracy, and also alleviate the dependence of Re-ID on the detection performance, which can effectively improve the tracking effect. Furthermore, adding ACM, MOTA increases from 77.2 to 81.1 and IDF1 increases from 80.3 to 82.4, which shows that the index is significantly improved when the three networks work together.

4.3. Comparison with State-of-the-Art Methods

In this section, we compare the performance of REACTrack with previous MOT methods on MOT16. As shown in

Table 2. MOT 16 Results. # indicates one-shot method. * indicates other joint detection and tracking methods which adopt non-ReID methods for data association. The best results are shown in bold.

Algorithm	MOTA ↑	IDF1 ↑	MT ↑	ML ↓	IDs ↓	FP ↓	HOTA ↑
* TubeTK [44]	66.9	62.2	39.0	16.1	1236	11,544	48.7
# JDE [24]	64.4	55.8	35.4	20.0	1544	10,642	_
* CTracker [19]	67.6	57.2	32.9	23.1	1897	8934	48.8
# FairMOT [25]	74.9	72.8	44.7	15.9	1074	10,163	_
# Ours	81.1	82.4	49.5	14.1	1012	8891	64.3

, REACTrack achieves the best results on a variety of metrics and outperforms the comparable methods compared by a wide margin, especially on the MOTA, HOTA, IDF1 and IDS metrics.

In order for our algorithm to be compared with ByteTrack, it will be based on FairMOT as shown in

Table 3. Comparison with ByteTrack based on FairMOT in MOT17. The best results are shown in bold.

	MOTA ↑	IDF1 ↑	IDS ↓	FP ↓	FN ↓
Motion (k) + Re-ID	69.1	72.8	299	1976	14,443
Motion (k) + EFGCN Re-ID	74.2	75.3	274	1857	13,658
Motion (k) + BYTE	70.3	73.2	236	2189	13,625
Motion (k) + ACM	73.4	74.8	221	2075	13,264

. In order to evaluate the effectiveness of EFGCN Re-ID and ACM, we compare it with ByteTrack using different patterns. The first row represents the tracking process embedded by FairMOT combined with Re-ID. The second row indicates the combination of FairMOT with our proposed EFGCN Re-ID. The MOTA is improved by 5.1% and the IDF1 is improved by 2.5%, so the features extracted by EFGCN are more effective, and it also reduces the dependence on the detection performance and improves the tracking effectiveness. The fourth row compares our ACM algorithm with BYTE. The MOTA improves from 70.3 to 73.4 and the IDF1 improves from 73.2 to 74.8, indicating that our proposed method improves the tracking ability because of the embedding of the EFGCN Re-ID in the correlation process as well as the in-depth processing judgments of the ACM for the occluded targets. The IDS decreases from 236 to 221, which indicates better robustness of tracking.

As shown in

Table 4. MOT 17 Results. # indicates one-shot method. * indicates other joint detection and tracking methods which adopt non-ReID methods for data association.

Algorithm	MOTA ↑	IDF1 ↑	MT ↑	ML ↓	IDs ↓	FP ↓	HOTA ↑
* Center Track [5]	67.8	64.7	34.6	24.6	3039	18,498	52.2
* Ctracker [19]	66.6	57.4	32.2	24.2	5529	22,284	49.0
* ByteTrack [7]	80.3	77.3	53.2	14.5	2196	25,491	63.1
# FairMOT [25]	73.7	72.3	43.2	17.3	3303	27,507	59.3
# CSTrack [34]	74.9	72.6	41.5	17.5	3567	23,847	59.3
Ours	80.5	80.1	52.1	14.3	2031	19126	62.8

, we have compared the performance of REACTrack with the previous MOT method on MOT17. In the one-shot method approach, REACTrack improved the IDF1 metric of FairMOT from 72.3 to 80.1, improves MOTA from 73.7 to 79.2, improves HOTA from 59.3 to 62.8, and reduces the IDs from 3303 to 2031. Compared with the BYTE, we can see that Simple Track improves IDF1 from 77.3 to 80.1 and reduces ID from 2196 to 2031, and the HOTA of both methods is also close. This shows that our tracking method is more effective than the JDE strategy, and the detector used does not want to pass compared to the BYTE strategy, and it also shows that our proposed prediction of the distance difference loss is effective for MOTA, and also the reduction of the ID value shows that our proposed EFGCN and trajectory correlation method can can improve the quality of the features, and reduces the number of switching of the IDs in the case of target occlusion.

4.4. Visualization Results

We show in Figure 5 that a number of scenarios that are prone to identity switching and contain sequences from MOT17. We used different tracking methods to generate the visualization results. It can be seen that REACTrack can effectively handle the identity switching problem due to tracking target occlusion. In addition, Figure 6 and Figure 7 show some tracking examples on the MOT17 dataset.

In the case of a moving picture, motion blur is likely to occur during tracking. At this point, it becomes crucial to guarantee the feature quality based on the basis. The method in this paper has used is EFGCN, which can guarantee the robustness of the pedestrian feature quality and the effectiveness of tracking under camera motion conditions.

5. Conclusions

In this paper, we have proposed a Predicted Distance Difference Loss, or PDDL, for the JDE-based MOT method to mitigate the detection of centroid offsets, and the PDDL can be easily applied to existing trackers to improve not only tracking performance but also detection accuracy. In addition, we have designed an EFGCN that extends the effective feature range for mitigating Re-ID’s dependence on detection performance and improving feature quality, and have proposed a trajectory correlation correction strategy, ACM, to minimize the loss of occluded targets. Together, these form REACTrack, which has achieved 62.8 HOTA and 80.1 IDF1 on the MOT17 dataset, ranking first among all JDE-based methods.

In future work, we will consider enhancing target features in the temporal dimension and designing an algorithm that can be combined with cross-camera, multi-object tracking and applied to a wider range of scenarios.