A Target Re-Identification Method Based on Shot Boundary Object Detection for Single Object Tracking

Abstract

With the advantages of simple model structure and performance-speed balance, the single object tracking (SOT) model based on a Transformer has become a hot topic in the current object tracking field. However, the tracking errors caused by the target leaving the shot, namely the target out-of-view, are more likely to occur in videos than we imagine. To address this issue, we proposed a target re-identification method for SOT called TRTrack. First, we built a bipartite matching model of candidate tracklets and neighbor tracklets optimized by the Hopcroft–Karp algorithm, which is used for preliminary tracking and judging the target leaves the shot. It achieves 76.3% mAO on the tracking benchmark Generic Object Tracking-10k (GOT-10k). Then, we introduced the alpha-IoU loss function in YOLOv5-DeepSORT to detect the shot boundary objects and attained 38.62% mAP_75:95 on Microsoft Common Objects in Context 2017 (MS COCO 2017). Eventually, we designed a backtracking identification module in TRTrack to re-identify the target. Experimental results confirmed the effectiveness of our method, which is superior to most of the state-of-the-art models.

1. Introduction

Visual single object tracking (SOT) is a basic computer vision task that refers to the detection, extraction, recognition, and tracking of the single object selected in the video sequence, which can realize an understanding of the object’s behavior [1]. According to the object specified in the first frame in the video, the SOT network predicts the location and size of the object in subsequent frames. Visual SOT has a wide range of application prospects in aspects such as video surveillance [2], automatic driving [3], and human–computer interaction [4]. It is also the foundation for other computer vision tasks [5].

With the development of SOT, many practical and efficient algorithms have emerged [6]. One type of SOT model is based on correlation filters, such as KCF [7], SRDCF [8], and ECO [9]. This kind of method’s idea is that the cyclic convolution operation is completed by a fast Fourier transform through the obtained correlation filter in the next frame to realize the positioning of the target center point [10]. Another type of model is based on the Siamese network, which is designed to transform the object tracking task into a feature-matching task between a given template image and candidate images [11]. Models such as SiamFC [12], DaSiamRPN [13], and SiamAttn [14] are implemented based on this idea. In recent years, other types of Transformer [15] based models, such as RGB-T [16], STARK [17], MixFormer [18], and OSTrack [19], have become one of the most popular SOT methods due to their advantages of simple model structure and performance–speed balance. Another fundamental task in computer vision, object detection, is closely related to object tracking. Object tracking networks are often used to track detected objects for long-term object detection [20]. Current object detectors based on deep learning have become mainstream models, mainly divided into two-stage detectors and single-stage detectors. The former uses a proposal mechanism to obtain image regions that are more likely to contain objects in order to achieve target prediction, mainly including R-CNN [21], Fast R-CNN [22], and Faster R-CNN [23]. The latter directly detects objects from anchors after extracting input image features, mainly including SSD [24], YOLO series [25], and DETR [26]. For example, the detectors of the YOLO series directly pre-define anchors with multiple scales at all locations of the image and predict the category probability and confidence score, greatly improving the speed of object detection [27].

However, there are some challenges that have been difficult to solve for SOT models, mainly including target occlusion, out-of-view, deformation, and scale variation. These challenges can easily result in an inability to continue tracking or result in tracking errors. Figure 1 shows some examples in the tracking benchmark dataset GOT-10k [28].

In fact, not only in SOT but also in other tasks of computer vision, such as multi-object tracking (MOT), object detection, and person re-identification (Re-ID), these four major challenges are urgent issues to be addressed. The specific descriptions of these challenges are as follows:

(1)

Target occlusion. Due to the complex environment in videos, when the tracked targets are moving or when other objects in the shots are moving, the tracked targets may be obscured by other objects for a period of time. In Figure 1, the moving ship is obstructed by the railing in front of the camera.

(2)

Target out-of-view. This situation often occurs in videos where the tracked targets are moving rapidly, regardless of whether the shots are fixed or moving. Because the targets are generally not aware of the range of the shots, as they move, it may cause them to leave the cameras. The monkey’s rapid movement causes it to leave the shot in Figure 1.

(3)

Target deformation. Tracked targets with irregular shapes and sizes or actively changing their own appearance can cause this situation, especially when they move. In Figure 1, the small boat changes its plane shape in the shots due to its rotational motion.

(4)

Target scale variation. This situation occurs when the tracked targets approach or move away from the shots. The small dog’s running back and forth in front of the camera changes its scale within the lens in Figure 1.

Even SOT models based on Transformers find it difficult to cope with these challenges, resulting in tracking errors. The target out-of-view in these challenges, which means that the object leaves the shot and reappears, is more likely to appear in videos than we imagine, making it harder for SOT models to be well applied to real life. Through our analysis, the main factors may be as follows:

(1)

There is no uniform standard for videos, which are shot from different perspectives, and it is common for objects to move in and out of view. Videos, especially daily videos, are often casually shot, capturing scenes that are not the entire actual space but a limited fan-shaped area. At the same time, due to the ease of interaction between objects in videos and the uncertainty of their behavior, it is easy to cause objects to leave the shot.

(2)

The SOT algorithms are generally weak in the re-identification of objects out of shots [29]. Most SOT models focus on enhancing the performance of tracking by improving the ability of feature extraction and relation modeling [19]. However, when objects are obscured or change their appearance, the effective features of the object itself are extremely limited [30]. After the target leaves the shot, the model will choose the object with the highest confidence score in the current frame for long-term tracking, causing an inability to achieve target re-identification in a short period of time.

Figure 2 shows the two possible reasons for poor tracking performance.

In attempting to address the problem of poor tracking results of target out-of-view, this paper proposes a target re-identification method based on shot boundary object detection for single object tracking, called TRTrack. Specifically, we build a bipartite matching model of candidate tracklets and neighbor tracklets optimized by the Hopcroft–Karp algorithm, which is used for preliminary tracking and judging the target leaves the shot. Then, TRTrack carries out object detection at the shot boundary by improved YOLOv5-DeepSORT, in which the original loss function is substituted with the alpha-IoU. The backtracking identification module in TRTrack crops the detected objects and inputs them back into the tracking model as the search region image. Finally, the target with high confidence score will be re-identified.

The contributions of this paper are shown below:

• To deal with the problem of unsatisfactory tracking results resulting from target out-of-view, we propose a target re-identification method based on shot boundary object detection for single object tracking, called TRTrack;
• We build a bipartite matching model of candidate tracklets and neighbor tracklets optimized by the Hopcroft–Karp algorithm to judge the target leaves the shot and introduce the alpha-IoU loss function to YOLOv5-DeepSORT to enhance object detection capability;
• Through a wide range of experiments by self-built videos dataset CLV and benchmark dataset, TRTrack is verified to be applied well for target re-identification in most video tracking tasks.

The rest of this paper is structured as follows: Section 2 presents the related work, including disappearing objects re-identification methods, a single object tracking model based on Transformer, and an object detection model following a top-down approach. Section 3 describes the proposed methodology of the target re-identification method TRTrack with a preliminary object tracking module, boundary object detection module, and backtracking identification module. Section 4 shows the detailed experiment and results. Section 5 concludes our work.

2. Related Work

2.1. Methods of Disappearing Target Re-Identification

The target re-identification mentioned in this paper is fundamentally different from a major task of computer vision, namely, person re-identification (Re-ID). Re-ID is geared to recognize the same person through videos obtained from different cameras [31]. It can be categorized into person detection, person tracking, and person retrieval [32]. The introduction of infrared technology has become a new development hotspot for Re-ID, and people have also begun to attach importance to infrared technology in other fields [33,34,35]. By contrast, the target re-identification in this paper only refers to the identification of targets that leaves the shot and reappears in SOT tasks.

Target out-of-view is essentially a special case of target disappearing, for which there are various algorithms that strive to resolve, especially for another special case, target occlusion. In fact, the target out-of-view is essentially the same as the target being obscured by the shot boundary.

The LMCF [36] model proposed by Wang et al. determines whether the object is obscured or disappears by observing the change degree of self-created metric APCE. Liu et al. [37] designed the BM Net with a multi-stream convolutional-LSTM network, which predicts the position of the target in subsequent frames based on its past trajectory. The Siam R-CNN [38] model uses the mechanism of object redetection to input the nearest object into the redetection network to judge and retrieve the object. Chen et al. [30] created the NeighborTrack, which uses the confidence score output from the object tracking model to automatically utilize the information of neighbor regions that are not obscured to re-identify the tracking target. For multi-object tracking, Ahn et al. [39] introduced an attention-based re-identification model, which extracts feature vectors from images to correlate objects based on their appearance effectively.

However, most of these methods rarely use the Transformer [15], leading to their lack of tracking performance and computation speed. They aim more at reducing the tracking errors caused by the object being obscured in the shots instead of re-identifying the object that leaves the shot and reappears.

2.2. Single Object Tracking Model Based on Transformer

Transformer stands out in object tracking after Dosovitskiy et al. [40] formally introduced the Transformer model into computer vision, especially in the field of human pose estimation [41] or head posed estimation [42].

SOT models based on Transformers can be divided into two- and one-stream frameworks [19]. The basic difference lies in how to carry out feature extraction and relation modeling on the template image and the search region image:

(1)

Two-stream framework. This framework first inputs the template and the search region into the backbone of the model and shares the weight. It then concatenates the output results to feed into the Transformer. Finally, the location of the object is predicted by classification, regression, and other methods. In recent years, Chen et al. [43] proposed TransT, which fuses iterative features through stacked self-concern layers and cross-concern layers. The STARK [21] model implemented by Yan et al. connects a new template with the search region in a way that automatically updates template images. Lin et al. [44] proposed SwinTrack based on the total attention mechanism instead of using CNN and other neural networks. These algorithms have satisfactory tracking accuracy, but their inference efficiency is not very fast because of heavy relation modeling.

(2)

One-stream framework. In this framework, the template and the search region are concatenated before they are input into the backbone, and the subsequent process is similar to that of the two-stream framework. A typical example is the MixFormer [18] developed by Cui et al., which introduces a mixed attention module to build an information interaction channel between the template-search image pairs. The OSTrack [19] proposed by Ye et al. connects the template and the search region to bidirectional information flows to combine feature learning with interaction. Chen et al. [45] built the SimTrack, which is a simplified tracking model using the Transformer as a backbone for relation modeling and feature extraction. These algorithms achieve not only high tracking accuracy but also fast inference speed, thereby balancing between performance and speed.

However, regardless of the two-stream framework or one-stream framework, most single object tracking methods based on Transformer focuses on enhancing the performance by improving the relation modeling and feature extraction capabilities and lack targeted solutions to common object tracking cases such as object occlusion and deformation [29]. When the object is obscured or changes its appearance, the effective features of the object itself are extremely limited [30]. Target out-of-view is a special case of target disappearing, so it is difficult to re-identify the target in a short time when it reappears.

2.3. Object Detection Method Following the Top-Down Approach

Object detection is a basic task in computer vision, with the purpose of identifying categories and predicting the position of objects in image sequences. It is widely used in fields such as pedestrian recognition [46], autonomous driving [47], and crop planting [48].

Object detection algorithms can be separated into top-down and bottom-up methods, with the main distinction being the period of holistic object generation and evaluation. The top-down approach is still the most commonly used method nowadays, including two-stage and one-stage methods:

(1)

Two-stage method. This method uses a proposal mechanism to decrease negative candidates generated by anchors and outputs the object detection results consisting of prediction bounding boxes and corresponding probabilities of object category through the detection network, such as CNNs. R-CNN [21], proposed by Donahue et al., combines region proposals with CNNs and can predict and partition objects by applying high-capacity convolutional neural networks. Girshick [22] designed the Fast R-CNN object detection model, which uses deep convolutional networks to classify object proposals efficiently; R-FCN [49], developed by Dai et al., contains position-sensitive score maps to resolve the contradiction between image classification and object detection. However, this method often results in a long training time and slow testing speed due to a large amount of repeated computation of convolutional features.

(2)

One-stage method. This method is created to realize object detection directly from anchors after extracting the input image features without using any proposal elimination mechanisms. Wei et al. [24] presented the SSD, which removes the generating proposal and image feature extraction modules and implements all the work of the model into a single network. RetinaNet [50] is designed to address imbalance problems of object detection by optimizing the standard cross entropy loss. Redmon et al. [25] converted the object detection task into a regression problem of object bounding boxes and corresponding category probabilities while proposing a single object detection network, YOLO, which has attained optimal accuracy and speed performance. This type of method can optimize the detection performance from end to end and has very high computation efficiency.

Regardless of whether a two- or a one-stage method is used, inherent problems arise in the object detection model, such as frequent ID switching of the detected object may result from occlusions. Therefore, extra models such as DeepSORT [20] can be used to improve object detection results. DeepSORT introduces deep learning into the SORT [51] algorithm and reduces identity switching by adding appearance descriptors.

3. Proposed Method

The overall architecture of TRTrack, a target re-identification method based on shot boundary object detection for single object tracking, is shown in Figure 3.

3.1. Preliminary Object Tracking Module

In the preliminary object tracking module, the base structure of OSTrack-384 [19] with powerful SOT performance-speed balance perfectly fits our goal, which is selected as the main component of the module. The post-processor NeighborTrack [30] is used to optimize the preliminary tracking results. It is a method that uses the confidence score output from the backbone of an object tracking model to automatically derive the information of neighbor regions that are not obscured.

The bipartite matching model in TRTrack maintains both the candidate pool $P^c$ and the neighbor pool $P^n$. The former is used to select the most suitable prediction tracking object, and the latter is used to verify the selected prediction tracking object. It converts the association problem between the template and search region in SOT into the bipartite matching problem between the candidate tracklets $S^c=P^c$ and the neighbor tracklets $S^n=P^n{\displaystyle \cup }\left\{\eta \right\}$, where $\eta =\left\{b_{t-1},\dots ,b_{t-\tau }\right\}$ represents the real prediction tracking boxes, $t$ is the sequence number of frames at any time, and $\tau$ is the number of frames to be traced. The weight $w_{ij}$ on the edge between two nodes, ${\xi }_i^t\in S^c$ and ${\zeta }_j^t\in S^n$, represents the average Jaccard overlap calculation between the two tracklets. This means that if ${\xi }_i^t=\left(b_{t-1}^c,\dots ,b_{t-\tau }^c\right)$ and ${\zeta }_j^t=\left(b_{t-1}^n,\dots ,b_{t-\tau }^n\right)$, the formula of the weight is denoted as

$$w_{ij}=\frac{1}{\tau }{\displaystyle \sum }_{k=t-\tau }^{t-1}IoU\left(b_k^c,b_k^n\right)$$

(1)

where $IoU$ represents the Jaccard overlap calculation between the two prediction bounding boxes, which reflects the similarity between the candidate tracklets $S^c$ and the neighbor tracklets $S^n$.

It is worth noting that the Hopcroft–Karp algorithm, which can achieve the maximum matching of the bipartite graph and has high computation efficiency, is introduced to implement bipartite matching. The time complexity of it is $O=\left(\left|E\right|\sqrt{\left|V\right|}\right)$, where $\left|E\right|$ is the number of edges of the bipartite graph and $\left|V\right|$ is the number of vertices in the bipartite graph. The specific formula of the improved bipartite matching result is as follows

$$ma{t}_t=HK\left(S^c,S^n\right)$$

(2)

where $ma{t}_t$ indicates the bipartite matching result of the $I_t$ frame. If $S^c$ does not match $S^n$, which means the candidate tracklet ${\xi }_m^t$ does not match the target tracklet $\eta$, it indicates that the target object is obscured or disappears. A function $at\text{\_}bdy$ is added to judge whether the prediction bounding box $b_t$ of the current frame $I_t$ is at the shot boundary. The equation is as follows

$$\begin{gathered} at\text{\_}bdy\left(b_t\right)=\left(x\in \left(0,\frac{W}{13}\right)\right)\cap \left(y\in \left(0,\frac{H}{13}\right)\right)\cap \left(x+w\in \left(\frac{12W}{13},W\right)\right) \\ \cap \left(y+h\in \left(\frac{12H}{13},H\right)\right) \end{gathered}$$

(3)

where $b_t=\left(x,y,w,h\right)$ are the bounding box containing abscissas and ordinates of the upper left corner and the size, $W$ is the width of the video, and $H$ is the length of the video. The final judgment of whether the target leaves the shot is shown as

$$ou{t}_t=\widetilde{ma{t}_t}\cap at\text{\_}bdy\left(b_t\right)$$

(4)

where $\widetilde{ma{t}_t}$ represents the inverse value of $ma{t}_t$, and $ou{t}_t\in \left\{0,1\right\}$ is the judgment variable of whether the target leaves the shot in the frame $I_t$. If the value is 1, the target has left the shot. Conversely, a value of 0 means that the target is still in the shot.

Figure 4 shows the feature maps obtained in the tracking network of the preliminary tracking module in TRTrack. The color in the feature maps reflects the similarity estimation between each position of the search region image and the target, and the greener the color indicates that the position of the current image is more likely to be the target.

3.2. Boundary Object Detection Module

In videos, the positions of the out-of-view targets returning from outside to inside the shots are uncertain, meaning that the target can reappear from almost any location at the shot boundary. Therefore, the YOLOv5-DeepSORT model, with excellent accuracy of object detection and inference speed, becomes a suitable tool for detecting the target in the boundary object detection module.

He et al. [52] utilized the power transformations to existing IoU loss functions, such as GIoU and CIoU, to develop a new IoU loss function, alpha-IoU. The $\alpha$ in the alpha-IoU represents a power parameter that provides greater flexibility for the detector to achieve different bounding box regression accuracy. Abundant experimental results reveal that the alpha-IoU can realize more accurate object detection by weighting the gradient and loss of objects with high IoU values in object detection models [53].

The original loss function of YOLOv5 is replaced by the alpha-IoU in the boundary object detection module of TRTrack to enhance the object detection accuracy and flexibility of the boundary object detection module. Based on the idea of alpha-IoU, we set the loss function only takes effect when the parameter $\alpha$ is greater than 1. The formula is as follows

$$L_{\alpha -IoU}=\frac{1-Io{U}^{\alpha }}{\alpha }, \alpha >1$$

(5)

where $IoU$ means the original Jaccard overlap calculation representing the intersection ratio of the prediction bounding box and the ground truth. In this case, the whole loss value of YOLOv5 in TRTrack is as follows

$$L_{total}=L_{conf}+L_{class}+L_{\alpha -IoU}{}^{\prime }$$

(6)

where $L_{conf}$, $L_{class}$, and $L_{\alpha -IoU}{}^{\prime }$ indicate the confidence loss, the classification loss, and the actual alpha-IoU loss, respectively. The specific alpha-IoU calculation equation is shown as

$$L_{\alpha -IoU}{}^{\prime }={\displaystyle \sum }_{i=0}^{S^2}{\displaystyle \sum }_{j=0}^B\left(1-L_{\alpha -IoU}\right)$$

(7)

where $S^2$ represents the number of image grids, and $B$ is the number of every grid anchor box in object detection.

On this basis, after the method in Section 3.1 is used to judge whether the target leaves the shot, the object detection is conducted at the shot boundary, and the detection range function is $at\text{\_}bdy$ mentioned in Section 3.1. Then, the results of object detection are shown as

$$det=\left\{d_1,\dots ,d_n\right\}\in at\text{\_}bdy\left(b_d\right)$$

(8)

which represents a collection of detection bounding boxes for objects at the shot boundary, and $b_d=\left(x,y,w,h\right)$ are the whole object detection bounding boxes.

3.3. Backtracking Identification Module

By taking advantage of the peculiarity of the SOT model in TRTrack that it transforms object tracking tasks into a feature extraction and matching problem between the template image and the search region image, we design the backtracking identification module, which is implemented on the object detection results $det=\left\{d_1,\dots ,d_n\right\}$.

For the objects detected at the boundary of the shot, TRTrack crops them out by setting the pixel values in the frame except for the region of objects as 0 or 255. That is, only the detected objects are retained in the current frame. The formula is as follows

$$crop={\displaystyle \sum }_{k=1}^n{d}_k,d_k\in det$$

(9)

Then, the cropped image $crop$ is input back to the SOT model and replaces the current frame image as the new search region image for tracking. Then the confidence score $S$, corresponding to each current prediction tracking bounding box $B$, will be output from the head of the tracking model. The confidence score $S$ can be used to judge whether the target returns to the shot. The formula for judging results is shown as

$$i{n}_t=\{\begin{array}{cc}1,& S\ge \lambda \\ 0,& S<\lambda \end{array}$$

(10)

where $i{n}_t\in \left\{0,1\right\}$ is the judgment variable of whether the target in the frame $I_t$ has returned to the shot, and $\lambda$ is the identification threshold parameter, which is set to 0.56 in this paper. Figure 5 shows some examples of using the backtracking identification module.

A flow-process diagram of the backtracking identification module is shown in Figure 6 to demonstrate its practical process better.

4. Experimental Results

The experimental platform is a Windows 11 64-bit system with two Nvidia GeForce RTX 4070Ti graphics cards, 64 GB running memory, Pytorch 1.10.0 deep learning framework, and Cuda 11.3 parallel computing platform, implemented on Python 3.8.

4.1. Experimental Setup

4.1.1. Datasets

We used two mainstream medium and long-term datasets of the SOT field, i.e., LaSOT [54] and GOT-10k [28], to evaluate the performance of the object tracking module in TRTrack. The LaSOT dataset consists of 1400 sequences, totaling over 3.5 million frames. Each sequence of it includes various challenges originating from the wild, such as target occlusion, deformation, etc. We also divided the training set and the test set according to the 4/1 ratio given by LaSOT. The GOT-10k dataset includes over 10,000 video segments with more than 1.5 million manually labeled bounding boxes. It contains a total of over 560 types of moving objects and 87 motion modes. We follow the split of GOT-10k, approximately 9.34 k of training data, and 420 test data in GOT-10k.

The benchmark dataset MS COCO 2017 was selected to evaluate our object detection model. MS COCO 2017 is an object detection dataset built by Microsoft, with a total of 80 object categories. The dataset contains objects of various sizes, some of which are noisy or obstructed, making it challenging. There are over 118 k and 5 k images in the training and testing sets, respectively. We follow this split in experiments.

For target re-identification, considering the target re-identification problem of objects in videos studied in this paper, we found that the classroom videos fit our research needs very well. In classroom videos, because the interaction between teacher and student is crucial in a lecture, the behavior of a teacher leaving the podium to communicate with students can easily lead to the teacher being out of view.

On this basis, we established a classroom video dataset CLV, which is selected from actual classroom videos with prominent classroom video features, such as targets’ frequently moving in and out of the shots. It includes 16 main classroom video segments, with more than 6400 manually labeled bounding boxes when the target leaves the shot and reappears. The video information of the dataset is shown in

Table 1. Basic information of classroom video dataset CLV.

Video ID	Video Duration	Video Size	Frame Rate	Number of Times the Target Leaves the Shot	Total Time of the Target Leaves the Shot
001	40:20	1920 × 1080	30 fps	11 times	79.674 s
002	50:35	1920 × 1080	30 fps	9 times	179.614 s
003	37:42	1920 × 1080	30 fps	11 times	106.721 s
004	33:51	1920 × 1080	30 fps	9 times	142.153 s
005	43:35	1920 × 1080	30 fps	24 times	332.490 s
006	45:14	1920 × 1080	30 fps	15 times	163.069 s
007	32:04	1920 × 1080	30 fps	3 times	110.833 s
008	41:40	1280 × 960	30 fps	8 times	58.689 s
009	38:26	1920 × 1080	30 fps	7 times	174.110 s
010	32:11	1920 × 1080	30 fps	14 times	88.457 s
011	36:58	1280 × 960	30 fps	6 times	101.776 s
012	40:46	1920 × 1080	30 fps	19 times	254.330 s
013	30:09	1280 × 960	30 fps	4 times	76.541 s
014	31:32	1920 × 1080	30 fps	13 times	94.879 s
015	35:27	1920 × 1080	30 fps	16 times	143.556 s
016	38:43	1920 × 1080	30 fps	12 times	167.123 s

.

4.1.2. Evaluation Metrics

We tested our preliminary tracking module using the evaluation metrics proposed by GOT-10k and LaSOT. For GOT-10k, including mean average overlap (mAO) and mean success rate (mSR). For LaSOT, including area under the curve (AUC), normalized precision (P_Norm), and precision (P). Three average precision (AP) and two mean average precision (mAP) in MS COCO 2017 were used to evaluate our object detection model.

For the self-built dataset CLV, we defined three metrics to evaluate the performance of target re-identification:

• $ET$

The unit is “times”, which represents the number of tracking errors using the SOT model in a video for tracking the target, mainly reflecting the stability of the model.

$$ET={\displaystyle \sum }_{k=1}^{n}1,T{e}_k\in TE$$

(11)

where $TE$ represents the total moments when tracking errors occur, and $T{e}_k$ means the moment when a tracking error occurs during the k-th tracking error process.

• $ED$

The unit is “second”, which represents the total duration of tracking errors by using the SOT model in a video for tracking the target, mainly reflecting the accuracy of the model.

$$ED={\displaystyle \sum }_{k=1}^n(T{c}_k-T{e}_k)$$

(12)

where $T{c}_k$ represents the moment when the tracking error was corrected during the k-th tracking error process.

• $TD$

The unit is “second”, which represents the total duration of the time interval between the target’s reappearance after leaving the shot and being tracked again by using the SOT model in a video, mainly reflecting the ability of target re-identification of the model.

$$TD={\displaystyle \sum }_{k=1}^n(T{r}_k-T{a}_k)$$

(13)

where $T{r}_k$ and $T{a}_k$ represent the moment when the tracker re-identifies the target and the moment the target reappears in the shot during the k-th target re-identification process, respectively.

4.1.3. Implementation Details

In this study, the OSTrack-384 [19] is selected as the main component of the preliminary object tracking module. The input sizes of templates and search regions are 192 × 192 pixels and 384 × 384 pixels, the initial learning rate was set to 4 × 10⁻⁵, the batch size was set to 128, and the training epoch was defined as 300.

The bipartite matching model in TRTrack based on NeighborTrack [30] is used to optimize the tracking results. For SoftNMS in it, the IoU threshold was set to 0.25, and the time period $\tau$ of backtracking tracklets was set to 9.

For YOLOv5-DeepSORT in the boundary object detection module of TRTrack, the YOLOv5n network with balanced performance is chosen as the basic module of object detection, which is introduced in the alpha-IoU loss function. Following the universal approach, the training batch size was valued at 32, the decay rate was set to 5 × 10⁻⁴, and the learning rate was defined as 1 × 10⁻². In particular, the value of parameter $\alpha$ of the alpha-IoU loss function is 3.

4.2. Results and Analysis

4.2.1. Results of Preliminary Object Tracking

As we mentioned in Section 4.1.1, two mainstream medium and long-term SOT benchmark datasets, LaSOT [54] and GOT-10k [28], were selected to evaluate the performance of our preliminary tracking module. LaSOT is a challenging long-term tracking benchmark. GOT-10k adopts a one-shot tracking rule, which requires only training the tracker on its training split, and the object classes between the train and test split do not overlap. We follow this rule to train our model. The comparison of our preliminary tracking module and state-of-the-art models tracking results on two SOT benchmarks, LaSOT and GOT-10k, are concluded in

Table 2. Comparison with state-of-the-art models on two SOT benchmarks: LaSOT and GOT-10k.

Model	LaSOT			GOT-10k
	AUC	PNorm	P	mAO	mSR50	mSR75
AiATrack	0.690	0.794	0.738	0.696	0.800	0.632
SwinTrack-B-384	0.702	0.784	0.753	0.724	0.805	0.678
MixFormer-L	0.701	0.799	0.763	0.756	0.8573 *	0.728
OSTrack-384	0.711	0.811	0.776	0.737	0.832	0.708
NeighborTrack-OSTrack	0.722	0.818	0.780	0.757	0.8572	0.733
Ours	0.731	0.814	0.787	0.763	0.854	0.739

* The bold numbers in the table represents the optimal value in the evaluation metrics.

.

It can be seen that, for LaSOT, compared with the best version of NeighborTrack, namely, NeighboTrack-OSTrack, the metric AUC and P of the preliminary object tracking module in TRTrack are better, which are 0.731 and 0.787, respectively. A similar situation also occurs in the results of GOT-10k, in which the metric mAO and mSR₇₅ of ours are higher than other state-of-the-art models, which are 0.763 and 0.739, respectively.

4.2.2. Results of Object Detection

For our object detection model, the benchmark dataset MS COCO 2017 was selected to test. The object detection results of different loss functions on MS COCO 2017 are demonstrated in

Table 3. Detection results of YOLOv5-DeepSORT in our method with different loss functions.

Loss Function	AP	AP50	AP75	mAP	mAP75:95
LIoU	47.61%	67.52%	53.48%	48.76%	34.51%
LGIoU	50.10%	69.49%	55.58%	51.22%	36.04%
LCIoU	49.38%	68.84%	54.69%	50.59%	35.58%
Ours	52.97% *	72.19%	57.84%	53.51%	38.62%

* The bold numbers in the table represents the optimal value in the evaluation metrics.

.

Compared with IoU, GioU, and CIoU, YOLOv5-DeepSORT, with the alpha-IoU loss function in our method, achieves impressive results. The metric AP, AP₅₀, AP₇₅, mAP, and mAP_75:95 are improved by 2.87%, 2.70%, 2.26%, 2.29%, and 2.58%, respectively. Such a high object detection accuracy of the model is easily estimated to be appropriate for the detection task of the shot boundary objects.

Figure 7 reveals the results of using our boundary object detection module to detect objects at the shot boundary in some classroom videos of the self-built dataset CLV.

4.2.3. Results of Target Re-Identification

The experimental content of this part was mainly using the proposed model TRTrack, a target re-identification method based on shot boundary object detection for SOT, and state-of-the-art models, including OSTrack-384 [19], MixFormer-L [18], and NeighborTrack-OSTrack [30], to conduct comparative experiments through the classroom video dataset CLV established in this paper.

For these SOT models, OSTrack-384 (proposed by Ye et al.) connects the template and the search region to bidirectional information flows to combine feature learning with interaction. MixFormer-L, developed by Cui et al., introduced a mixed-attention module to build an information interaction channel between the template–search image pairs. NeighborTrack-OSTrack uses the confidence score to automatically utilize the information of neighbor regions that are not obscured. The above three models all achieve good performance in SOT tasks.

The experimental results can reflect the performance of TRTrack for target re-identification, which is shown in

Table 4. Target re-identification results of the CLV dataset.

Video ID	OSTrack-384			MixFormer-L			NeighborTrack-OSTrack			TRTrack (Ours)
	ET (Times)	ED (s)	TD (s)	ET (Times)	ED (s)	TD (s)	ET (Times)	ED (s)	TD (s)	ET (Times)	ED (s)	TD (s)
001	20	44.62	13.98	26	49.83	18.65	9	25.42	24.49	2 *	2.72	2.64
002	15	27.93	8.72	24	32.11	14.56	5	430.76	416.88	2	2.71	2.71
003	16	34.77	14.21	13	24.89	17.55	6	29.56	27.14	4	5.27	5.27
004	19	41.10	17.44	19	29.74	18.33	8	24.11	23.98	4	2.94	2.94
005	22	137.73	95.56	26	113.86	106.49	5	38.66	36.54	3	3.01	3.01
006	8	146.29	136.61	12	90.01	74.79	4	9.12	7.55	2	2.01	1.89
007	17	24.80	8.56	8	17.13	6.59	0	0.00	0.00	0	0.00	0.00
008	9	13.11	6.78	6	14.89	4.23	1	1.29	1.29	0	0.00	0.00
009	14	47.88	24.96	12	39.47	26.55	11	26.54	24.48	3	5.42	4.89
010	7	16.22	14.56	9	21.54	17.77	4	5.89	5.43	2	1.69	1.43
011	16	42.98	36.66	17	54.32	47.62	1	4.32	4.16	2	5.67	4.93
012	24	145.96	115.43	22	121.11	103.67	13	254.55	241.69	7	9.43	9.12
013	9	54.22	38.14	10	43.12	34.56	6	16.55	14.29	3	8.76	7.84
014	12	15.77	12.97	11	18.56	12.13	3	3.98	3.68	0	0.00	0.00
015	14	66.35	38.54	16	64.59	42.84	8	54.51	48.67	2	7.56	7.07
016	15	101.56	87.52	18	121.76	105.43	7	43.56	41.70	4	11.56	8.79
Avg	14.81	60.08	41.92	15.56	53.56	40.74	5.69	60.55	57.62	2.50	4.30	3.91
Sum	237	961.29	670.64	249	856.93	651.76	91	968.82	921.97	40	68.75	62.53

* The bold numbers in the table represents the optimal value in the evaluation metrics.

.

The table shows that compared with OSTack-384, MixFormer-L, and NeighborTrack-OSTrack, the TRTrack model proposed in this paper has an excellent performance in the classroom video dataset CLV, and the metric ET, ED, and TD are almost optimal in 16 videos.

The average value of ET is 2.5 times, and the sum value of it is 40 times, which is less than 3.19 times and 51 times of the second-best model NeighborTrack-OSTrack, indicating that TRTrack possesses good tracking stability and is not prone to tracking errors. The average value of ED is 4.3 s, and the sum value is 68.75 s, which is an order of magnitude less than other popular algorithms, indicating that TRTrack has a high tracking accuracy. The metric TD contains the average and sum value of the time interval between the target’s reappearance after leaving the shot and being tracked again in a video. For other tracking algorithm models, TD has the order of tens of seconds and hundreds of seconds, accounting for most of the tracking error duration, namely the metric ED. This is enough to show that the re-identification of the target after leaving the shot is a crucial object-tracking problem, and the poor ability in this aspect of current SOT models is the major source of tracking errors. The average value of the TD of TRTrack is 3.91 s, and the sum value is 62.53 s, indicating the effective target re-identification ability of our method.

The comparison between the target re-identification performance of TRTrack and other algorithms is shown in Figure 8.

4.2.4. Ablation Study of Target Re-Identification

To reflect the optimization results of several modules in TRTrack, including the bipartite matching model in the preliminary object tracking module and improved YOLOv5-DeepSORT in the boundary object detection module, extensive ablation experiments were carried out.

• Effect of the bipartite matching model of the preliminary object tracking module

We used the original NeighborTrack and the bipartite matching model in TRTrack to conduct comparative experiments in our own classroom video dataset CLV to demonstrate the effect of optimizing the NeighborTrack. A part of the experimental results is shown in

Table 5. Results of the bipartite matching model of the preliminary object tracking module.

Video ID	TRTrack with Original NeighborTrack			TRTrack with Bipartite Matching Model (Ours)
	ET (Times)	ED (s)	TD (s)	ET (Times)	ED (s)	TD (s)
001	4	8.74	6.79	2 *	2.72	2.64
002	5	9.26	7.42	2	2.71	2.71
003	4	6.38	5.16	4	5.27	5.27
004	5	7.29	6.94	4	2.94	2.94
005	7	12.16	10.07	3	3.01	3.01
006	6	6.11	4.83	2	2.01	1.89
007	0	0.00	0.00	0	0.00	0.00
008	1	1.65	0.54	0	0.00	0.00

* The bold numbers in the table represents the optimal value in the evaluation metrics.

.

Table 5 shows that the bipartite matching model can enhance the ability of target re-identification of TRTrack. In most videos of the CLV dataset, the metric ET, ED, and TD of TRTrack with the bipartite matching model are superior to TRTrack with the original NeighborTrack.

• Effect of improved YOLOv5-DeepSORT of boundary object detection module

Apart from comparing the effects of different loss functions on object tracking, the original YOLOv5-DeepSORT and our improved YOLOv5-DeepSORT are used to conduct a comparative experiment, which is also based on the classroom video dataset CLV to demonstrate the optimized effect. A part of the experimental results is shown in

Table 6. Results of improved YOLOv5-DeepSOT of boundary object detection module.

Video ID	TRTrack with Original YOLOv5-DeepSORT			TRTrack with Improved YOLOv5-DeepSORT (Ours)
	ET (Times)	ED (s)	TD (s)	ET (Times)	ED (s)	TD (s)
001	7	17.53	15.44	2 *	2.72	2.64
002	8	16.74	13.10	2	2.71	2.71
003	7	11.98	10.56	4	5.27	5.27
004	9	19.24	15.13	4	2.94	2.94
005	14	22.87	19.06	3	3.01	3.01
006	7	14.88	12.65	2	2.01	1.89
007	4	5.43	3.29	0	0.00	0.00
008	5	8.47	7.08	0	0.00	0.00

* The bold numbers in the table represents the optimal value in the evaluation metrics.

.

The effect of improving the loss function is obvious. Table 6 shows that the target re-identification ability of TRTrack based on YOLOv5-DeepSORT with the alpha-IoU loss function is significantly better than TRTrack with the original YOLOv5-DeepSORT. The metric ET, ED, and TD have apparent optimization, indicating that different loss functions have different impacts on the effectiveness of models.

4.2.5. More Visualization of Target Re-Identification

Apart from classroom videos, our target re-identification model, TRTrack, also has excellent target re-identification results on publicly available SOT benchmark datasets. Figure 9 shows the target re-identification effect of TRTrack when some tracking objects leave the shots and reappear in the GOT-10k dataset, which reflects the target re-identification ability of TRTrack. It is worth noting that our model differs from other tracking algorithms in that when the target is identified as leaving the shot, tracking is no longer performed, and no prediction tracking bounding box is provided.

5. Conclusions and Future Work

5.1. Conclusions

In this paper, a target re-identification method called TRTrack based on shot boundary object detection is proposed to solve the issue of poor single object tracking due to target out-of-view. First, we build a bipartite matching model of candidate tracklets and neighbor tracklets optimized by the Hopcroft–Karp algorithm, which is used for preliminary tracking and judging the target leaves the shot. If the target is in the shot, the tracking will continue. Otherwise, the boundary object detection module based on improved YOLOv5-DeepSORT with the alpha-IoU loss function will detect the shot boundary objects in the video. Finally, the backtracking identification module in TRTrack will crop the detected objects out, replace the current frame image with them as the search region image and input them back into the object tracking model. According to the confidence score of tracking results, the target can be re-identified in a very short time.

The experimental results show that the preliminary object tracking model in our method achieves 73.1% AUC on LaSOT and 76.3% AO on GOT-10k. The YOLOv5-DeepSORT of the boundary object detection module obtains 38.62% mAP_75:95 on MS COCO 2017. The proposed target re-identification method TRTrack demonstrates its target re-identification ability at the self-built dataset CLV, which is superior to most state-of-the-art models and has practical application value.

5.2. Future Work

Although our target re-identification method TRTrack has achieved good results, some problems remain. For example, even though we used OSTrack-384 [19] and YOLOv5 with a good performance-speed balance in our model, the real-time performance of our approach is still not good enough. The current frame rate is about 10 fps, which can still be further improved. In future studies, we are expected to further optimize our model by introducing network pruning, model lightweight and other methods so as to improve the rate of target re-identification while maintaining accuracy. We look forward to better development of target re-identification for object tracking in the future.