ParallelTracker: A Transformer Based Object Tracker for UAV Videos

Abstract

Efficient object detection and tracking from remote sensing video data acquired by unmanned aerial vehicles (UAVs) has significant implications in various domains, such as scene understanding, traffic surveillance, and military operations. Although the modern transformer-based trackers have demonstrated superior tracking accuracy, they often require extensive training time to achieve convergence, and the information from templates is not fully utilized in and integrated into tracking. To accelerate convergence and further improve tracking accuracy, we propose an end-to-end tracker named ParallelTracker that extracts prior knowledge from templates for better convergence and enriches template features in a parallel manner. Our core design incorporates spatial prior knowledge into the tracking process through three modules: a prior knowledge extractor module (PEM), a template features parallel enhancing module (TPM), and a template prior knowledge merge module (TPKM). These modules enable rich and discriminative feature extraction as well as integration of target information. We employ multiple PEM, TPM and TPKM modules along with a localization head to enhance accuracy and convergence speed in object tracking. To enable efficient online tracking, we also design an efficient parallel scoring prediction head (PSH) for selecting high-quality online templates. Our ParallelTracker achieves state-of-the-art performance on the UAV tracking benchmark UAV123, with an AUC score of 69.29%, surpassing the latest OSTrack and STARK methods. Ablation studies further demonstrate the positive impact of our designed modules on both convergence and accuracy.

1. Introduction

Visual object tracking (VOT) [1,2,3,4,5,6,7] is a fundamental task in remote sensing and computer vision that involves predicting the position and shape of a target in each subsequent frame based on its position and shape in an initial frame. This task has numerous applications, including aerial or satellite platform surveillance, human–computer interaction [8], mobile robots [9], and autonomous driving [10]. Despite the ability of mainstream tracking algorithms to balance accuracy and speed [11,12,13,14,15,16,17], the task of object tracking faces significant challenges, such as fast-moving targets, background interference, occlusion, and changes in shape, as well as loss of targets. Therefore, designing a robust, accurate, and real-time tracker remains a daunting challenge. VOT tasks generally consist of a multi-level pipeline [17,18,19] that comprises three parts. The first part is feature extraction, which involves generic feature extraction from both the template and search region. The second part is information interaction, which aggregates the feature information of the template and the search region. The final part involves bounding box and location estimation, typically achieved through a specific task head that allows for precise target positioning and bounding box estimation.

The classic VOT algorithms, including correlation-based methods such as SiamFC (Siamese Fully Convolutional Network) [20], SiamRPN (Siamese Region Proposal Network) [21], C-RPN (Cascaded Region Proposal Network) [22], SiamFC++ [17], SiamBAN (Siamese Box Adaptive Network) [23], SiamCAR (Siamese Classification and Regression network) [24], and Ocean (Object-aware Anchor-free Network) [13], as well as online learning algorithms such as MOSSE (Minimum Output Sum of Squared Error filter) [25], KCF (Kernelized Correlation Filter) [26], CSR-DCF (Channel Spatial Reliability-Discriminative Correlation Filter) [27], ATOM (Accurate Tracking by Overlap Maximization) [28], DiMP (Discriminative Model Prediction) [29], FCOT (Fully Convolutional Online Tracking) [14], and ECO (Efficient Convolution Operators) [30], generally utilize CNN as the basic component of feature extraction and information interaction. However, CNNs are not always effective in modeling long-range dependencies between image content and time-series features, which may result in target loss in some cases (e.g., when the target is briefly occluded). In sequence modeling, a transformer [31] overcomes this problem with its global and dynamic modeling capabilities. For tracking tasks [32,33], the transformer can extract discriminative spatial and temporal features, making it a fundamental component of modern high-precision trackers. Although the flexibility of attention in vision transformers (VIT) [34,35] enables higher performance, one of the main challenges of current VIT-based trackers [19] is their slow training convergence speed due to the large amount of input image content, which places a heavier burden on computing resources [32].

The current algorithms used for tracking unmanned aerial vehicles (UAVs) face several challenges, as highlighted in previous studies [36,37]. These challenges can be summarized into two points. First, UAV videos are prone to motion blur, occlusion, and background clutter due to the rapid movement of the drone. The altitude variation of the drone’s six degrees of freedom (6 DOF) movement further complicates the task by causing objects to appear at different scales. As a result, extracting stable features from UAV videos is a comparatively difficult task compared to videos from other fields. Second, object tracking in UAV videos requires the rapid and accurate location of moving objects, making real-time performance a critical requirement. Therefore, it is necessary to reduce the convergence difficulty of a popular tracker for being deployed on the UAV platform. To address the challenges faced by existing tracking methods, we design a new tracker called ParallelTracker, which offers several key advantages. First, we design the prior knowledge extractor module (PEM) to extract spatial information from the template image, addressing the lack of spatial prior information in the VIT architecture. This spatial knowledge is utilized to assist the tracker in both training and inference while reducing the computational burden during training and maintaining optimal performance. Second, we design the template features parallel enhancing module (TPM), which is a parallel attention module that utilizes prior knowledge from the template image to capture more target-oriented discriminative features and enhance feature extraction. Furthermore, we design the template prior knowledge merge module (TPKM) based on TPM to further extract features and facilitate information communication between the template and search regions. TPKM utilizes position self-attention and channel self-attention to extract deep features from both the template and search region in parallel, allowing for information interaction in both position dimension and channel dimension through parallel attention operations. This improves the acquisition of correlations between the template and the search region. Finally, we use a simple location prediction head to complete the tracking process. Additionally, we design a score prediction head based on parallel attention to implement the target template update strategy. Our ParallelTracker requires fewer training epochs than the existing VIT-based trackers [18,19,38], effectively decreasing the difficulty of convergence and striking a balance between performance and speed.

The main contributions of this paper can be divided into two aspects. First, we propose a simple, concise, and effective VIT-based tracker called ParallelTracker that aims to enhance the accuracy of VIT-based tracking algorithms for UAV video by addressing the challenges of motion blur, camera motion, occlusion, and the convergence speed. We design PEM, TPM, and TPKM to assist feature extraction and information interaction, leading to faster convergence and better performance. Second, We demonstrate that ParallelTracker achieves state-of-the-art performance on the UAV tracking benchmark UAV123, faster convergence speed, and a real-time performance of 25 FPS on an NVIDIA GeForce RTX 2070 GPU.

2. Related Literature

Early Siamese tracking networks, such as SiamFC [20], SiamRPN [21], and SiamBAN [23], first extract features from the template and the search region using CNN networks with identical structures and parameters. The template and the search region features are then fused using correlation calculation to estimate the subsequent state. These networks also utilized CNN networks, including ResNet [39] and others [40], as the backbone for extracting image features.

The transformer [31], originally proposed for machine translation tasks in NLP (natural language processing), is now widely adopted for various vision tasks. It introduced a multi-head attention mechanism to enable information interaction among the elements of a sequence, thereby endowing it with unique memory and global computing capabilities. Compared to a CNN with a local receptive field, the core component of the transformer, namely self-attention, is capable of capturing global information and computing interdependencies among image features. The DETR model [41] introduced the transformer into object detection and surpassed many CNN-based methods. Many researchers [19,32,36] have also introduced DETR into the field of visual object tracking, achieving excellent results. OSTrack [18] proposed a complete end-to-end tracker that used the transformer to perform both feature extraction and information interaction simultaneously. TrDiMP [33] integrated transformer structures into tracking frameworks by separating the encoder and decoder of the transformer and exploring temporal information between frames, which effectively enhanced the robustness of Siamese-like trackers. TrTr [42] utilized contextual information extracted from the learned template image in the encoder and transmited it to the decoder for information exchange with the search region. TransT [32] proposed an attention mechanism for nonlinear feature fusion, which effectively integrated global information from both template images and search regions by capturing associated information from long-range features. STARK [19] concatenated template images and search regions to learn spatiotemporal features jointly across frames. In this study, ParallelTracker differs fundamentally from existing trackers in several aspects. Specifically, instead of using self-attention as in Dual_VIT [43], our designed module employs a parallel attention structure in feature extraction and information interaction. Moreover, we adopt multiple templates and search regions as inputs and utilize corner-based localization heads to generate bounding boxes.

3. Methodology

We propose an end-to-end tracking framework named ParallelTracker, which comprises the PEM module for extracting the spatial prior of the template, as well as the TPM and TPKM modules for enhancing the features and fusing the target information, respectively. In addition, ParallelTracker incorporates a corner-based fully convolutional localization head that estimates the bounding box of the tracked object. Moreover, we have introduced a confidence-based score head, which offers a specially designed mechanism for updating the online template to address potential deformations of the tracked object. The overall architecture of ParallelTracker is illustrated in Figure 1.

3.1. PEM, TPM and TPKM

We design PEM, as shown in Figure 2a, to extract prior knowledge from object templates.

We propose PEM to extract local spatial information from the template image, boosting the capacity of the tracker while mitigating the computational burden during training. The utilization of local spatial information in the vision transformer enables the model to capture specific local features of the object. Additionally, this approach imbues the model with inductive biases that are similar to CNNs, such as locality and spatial invariance. Prior knowledge can be defined as the local spatial feature of the target, which informs the model’s understanding of the target’s spatial context. The specific operation of PEM is as follows. First, we obtain downsampled template tokens by using a fully convolutional patch embedding layer. The patch embedding layer is composed of convolutional layers to increase the channel resolution while reducing the spatial resolution. The specific operation of patch embedding layers is: Given T templates (i.e., the first template and $T-1$ online templates) with the size of $T\times h_t\times w_t\times 3$, we first map it into patch embeddings using a convolutional layer with stride 4 and kernel size 7. Then, we flatten the patch embeddings, resulting in a token sequence with a size of $\frac{h_t}{4}\times \frac{w_t}{4}\times C$, where C represents the number of channels and is set to 96. Here, $h_t$ and $w_t$ denote the height and width of the template, respectively, and are both set to 128. Next, these tokens are fed into an average pooling layer, which reduces their resolution while preserving vital information. The target query is initialized using a random vector and upsampled through bicubic interpolation to match the shape of the template tokens. To map the template tokens and target query to the same linear space while retaining the spatial information of the template in the target query using a self-attention mechanism, we incorporate a linear mapping (LM) layer into our model. This allows us to extract local spatial information as spatial prior knowledge, which can guide subsequent modules in feature extraction, feature enhancement, and information interaction. As a result, it is possible to incorporate prior information, called prior tokens (PT), into the tracking network. This is a special feature of our model and sets it apart from other approaches in the field. As shown in (1),

$$\begin{array}{c}\hfill q_{Tq}=k_{Tt}=Linear\left(T_q\right)\\ \hfill v_{Tt}=Linear\left(T_t\right)\\ \hfill PT=Attention(q_{Tq},k_{Tt},v_{Tt})& \hfill =Softmax\left(\frac{q_{T_q}{k}_{T_t}}{\sqrt{d}}\right)v_{T_t}\hfill \end{array}$$

(1)

where d represents the dimension of the key, $T_q$ represents the target query, and $T_t$ represents the template tokens. The learnable target token $T_q$ is interpolated to match the spatial shape of the template tokens $T_t$, and then all tokens are linearly mapped to obtain $q_{Tq}$, $k_{Tt}$, and $v_{Tt}$. We utilize a self-attention mechanism to extract spatial prior information from the template tokens and fuse it into the target tokens, resulting in the generation of tokens with template prior knowledge.

TPM is designed to enrich template features, as illustrated in Figure 2b. TPM consists of two parallel attention modules and two feed-forward layers (FFN), with layer normalization (LN) applied before the FFN and LM layers. The TPM module utilizes prior tokens (PT) extracted by the PEM module to enhance the template features, thereby improving the ability of the extracted features to attend to both spatial and semantic information in the target template. The first attention module is utilized to enhance the spatial representation in the template tokens, while the other attention module effectively extracts key information from the template features and stores it in the enhanced prior tokens. The implementation of these two attentions is shown in Equation (2).

$$\begin{array}{cc}\hfill P{T}_{enhanced}& =FFN\left(LN\left(Attentio{n}_{PT}(PT,T_t)\right)\right)\hfill \\ \hfill T_t& =FFN\left(LN\left(Attentio{n}_{PT}\left(T_t\right)\right)\right)\hfill \end{array}$$

(2)

TPKM, as illustrated in Figure 2c, closely resembles the TPM module. TPKM consists of two multi-head attention layers and three feed-forward layers (FFN). Layer normalization is applied before the FFN and attention. Parallel attention is utilized to engage with distinct sequence feature dimensions, thereby intensifying the fused feature information communication, as shown in (3):

$$T,S=FFN\left(LN\left(split(Attentio{n}_{position}\left(LN\left(x\right)\right)+Attenti{n}_{channel}\left(LN\left(x\right)\right))\right)\right)$$

(3)

T and S denote the feature sequence of the template and the search region, respectively. The fused features are represented by x, which are obtained by concatenating online template/template tokens and search tokens along the same dimension. To perform information interaction, we employ parallel position attention and channel attention.

3.2. Detailed Structure of ParallelTracker

The proposed end-to-end ViT-based tracking network, ParallelTracker, embedded with PEM, TPM, and TPKM (as illustrated in Figure 2) consists of three parts: a backbone, a localization head, and a confidence prediction head. By integrating feature extraction and information interaction, our tracker achieves faster convergence speed. Moreover, it decouples the image content features from prior knowledge, without requiring any additional post-processing or integration modules.

Backbone. The backbone employs a step-by-step multi-block architecture strategy, which consists of three distinct blocks. These blocks operate on feature maps that have been down-sampled to the same scale, with the same number of channels. Each block varies slightly in structure, incorporating overlapping patch embedding layers and a different number ($N_i$) of either PEMs, TPMs, or TPKMs.

In our approach, we input T templates with a size of $T\times h_t\times w_t\times 3$ and a search region with a size of $h_s\times w_s\times 3$ into the tracker. The T templates comprise templates generated from the initial frame and $T-1$ online templates selected from the confidence prediction head. In this study, we set $T=2$. Additionally, the search region is cropped according to the previous target state. To map the input image to tokens and increase the channel dimension while reducing the spatial resolution of image features, we utilize the patch embedding layer, which comprises a convolutional layer, batch normalization, and ReLU. Subsequently, in the first block, the image features are converted into template tokens and search tokens with shapes of $\frac{h_t}{4}\times \frac{w_t}{4}\times C$ and $\frac{h_s}{4}\times \frac{w_s}{4}\times C$, respectively, through the embedding layer. Where C is 96, $h_t$ and $w_t$ are 128, $h_s$ and $w_s$ are 320. The template tokens are then fed into the PEM to obtain the template of prior knowledge.

In the second block, the image features are further down-sampled to $\frac{h_t}{8}\times \frac{w_t}{8}\times 3C$ and $\frac{h_s}{8}\times \frac{w_s}{8}\times 3C$, respectively, using the embedding layer. To enhance the template token features, we employ TPM in the next step. In the final block, the down-sampled features from the first two blocks are concatenated to generate the fused feature token sequence with a size of $(T\times \frac{h_t}{16}\times \frac{w_t}{16}+\frac{h_s}{16}\times \frac{w_s}{16})\times 5C$. We employ TPKM to interact information. Before being fed into the location head, the fused feature tokens sequence is first split into template tokens, online template tokens, and search tokens. However, only the search tokens are inputted to both the localization head and the score head. The search tokens are converted into the format of $\frac{h_s}{16}\times \frac{w_s}{16}\times 5C$. In this context, $h_s$ and $w_s$ indicate the spatial dimensions of the search region, whereas $h_t$ and $w_t$ indicate the spatial dimensions of the template. Unlike other trackers [36,38], we do not employ a multi-scale feature aggregation strategy.

Head. The proposed tracking network employs a fully convolutional localization head based on corners for predicting the bounding box, which bears a resemblance to the STARK method [19]. The localization head exclusively relies on a basic convolutional network to predict the upper-left and lower-right corners, after which it computes the expected probability distribution of the corners to predict the bounding box. Utilizing the localization head to estimate the bounding box represents Stage 1 of our tracking system. The effectiveness of tracking is significantly impacted by low-quality online templates. In view of this, we draw inspiration from the scoring modules in contemporary trackers [19,32,36] and propose our parallel scoring prediction head (PSH) for selecting reliable online templates based on expected confidence scores, as shown in Figure 3.

The PSH model is composed of three attention blocks, including two position attentions and one channel attention, and a three-layer perceptron. Initially, a learnable score token is utilized as a query to attend the search ROI tokens, allowing the score tokens to encode the extracted target information. The attended score token compares the extracted target with the tracked target at all positions of the initial target, and the scores are generated using MLP layers and sigmoid activations. Any online template prediction scores below 0.5 are deemed negative. The utilization of PSH to derive scores represents Stage 2 of our tracking process.

3.3. Training and Inference

Training. The training procedure of ParallelTracker generally adheres to the standard training protocol employed by current trackers [19]. We begin by pre-training the backbone network of ParallelTracker using the Dual_VIT model. Subsequently, we fine-tune the tracking network using labeled tracking datasets. We employ a combination of L1 loss and GIOU [44] loss as the objective function.

$$L_{loc}={\lambda }_{L_1}{L}_1(B,B^{\prime })+{\lambda }_{GIoU}{L}_{GIoU}(B,B^{\prime })$$

(4)

where ${\lambda }_{L_1}=5$ and ${\lambda }_{GIoU}=2$ are the weight values of the two losses; B is the ground-truth bounding box of the target, and $B^{\prime }$ is the predicted regression box.

Template online update. The template is an image patch of size $128\times 128$, which is cropped from the initial frame of a video. The online templates are also image patches of the same size, but they are dynamically selected from the current frames by PSH. Online templates play a crucial role in extracting target-related information from time series data. Nevertheless, tracking may encounter issues such as motion blur or objects going beyond the field of view, which may decrease the similarity between online templates and targets. To address this issue, we propose parallel scoring prediction head (PSH), as illustrated in Figure 3, to select reliable online templates using confidence scores. We train PSH separately using standard cross-entropy loss:

$$L_{score}=y_{i}log\left(p_i\right)+(1-y_i)log(1-p_i)$$

(5)

where y is the ground-truth label, p is the predicted confidence score, and i represents the ith sample.

4. Experiments

4.1. Implementation Details

Our tracker is implemented using Python 3.7 and PyTorch 1.7.1 and is trained on two A6000 GPUs. Notably, ParallelTracker is a compact tracker that does not utilize post-processing, position embedding, or multi-layer feature aggregation strategies.

Structure. As shown in

Table 1. The architecture for ParallelTracker. The input is a tuple of templates with shape of $128\times 128\times 3$ and search region with shape of $320\times 320\times 3$; S and T represent for the search region and template; $H{D}_i$ is the head number and embedding feature dimension in $Bloc{k}_i$; $C_i$ is the feature dimension of output features; $R_i^x$ and $R_i^z$ are the feature dimension expansion ratio in the MLP layer.

	Output Size	Layer Name	ParallelTracker
$Bloc{k}_1$	S: $80\times 80$ T: $32\times 32$	Patch Embedding	Downsampling rate 4
	S: $80\times 80$ T: $32\times 32$	PEM TPM	$$\left[\begin{array}{c}H{D}_1=1\\ C_1=96\\ R_1^x=8\\ R_1^z=4\end{array}\right]\times 3$$
$Bloc{k}_2$	S: $40\times 40$ T: $16\times 16$	Patch Embedding	Downsampling rate 2
	S: $40\times 40$ T: $16\times 16$	TPM	$$\left[\begin{array}{c}H{D}_2=4\\ C_2=128\\ R_2^x=8\\ R_2^z=4\end{array}\right]\times 4$$
$Bloc{k}_3$	S: $20\times 20$ T: $8\times 8$	Patch Embedding	Downsampling rate 2
	S: $20\times 20$ T: $8\times 8$	TPKM	$$\left[\begin{array}{c}H{D}_3=10\\ C_3=320\\ R_3^x=4\\ R_3^z=2\end{array}\right]\times 15$$
MACs		26.76 M
FLOPs		26.03 G
Speed (2070)		25 FPS

, we instantiate ParallelTracker with different numbers of PEM, TPM, and TPKM modules, as well as hidden feature dimensions at each block. We initialize the backbone of ParallelTracker using the Dual_VIT [43] model, which has been pre-trained on the ImageNet-1k dataset.

Training details. The training dataset comprises the train splits of several benchmark datasets, including TrackingNet [45], LaSOT [46], GOT-10k [47], and COCO [48], which are also used by other VIT-based trackers during training. More specifically, the training dataset includes 118,287 images from COCO2017, 10,000 video sequences from GOT-10k, 1120 video sequences from LaSOT, and 10,214 video sequences from the TrackingNet dataset. The sizes of search images and templates are $320\times 320$ pixels and $128\times 128$ pixels, respectively, corresponding to 25 and 4 times the bounding box area. The complete training process for the ParallelTracker consists of two stages. In the first stage, we train the backbone and localization head for 200 epochs, followed by an additional 40 epochs for training the confidence prediction head. We train ParallelTracker using ADAM [49] with weight decay. The learning rate is initially set to $1\times {10}^{-4}$ and decreases to $1\times {10}^{-5}$ after 100 epochs. To augment the data, we employ horizontal flipping and brightness jittering.

Inference. For ParallelTracker, we utilize the template generated from the initial frame, online templates selected by PSH, and the search region as inputs. By default, the online templates are updated every 200 frames during the tracking process. We update the online template with the image that has the highest confidence prediction score.

4.2. Comparison with State-of-the-Art Trackers

We validate the performance of ParallelTracker on a public UAV dataset, UAV123 [50], as the main experiment. We also report our results on a close-range dataset, LaSOT [46], to demonstrate the generalization ability of the method.

Evaluation Metrics. This study employs the one-pass evaluation (OPE) standard, as defined by prior literature [45], which comprises three key components: precision, normalized precision, and success rate. These metrics are commonly used to evaluate the tracking performance of visual trackers in the field of computer vision. The success rate is calculated by measuring the intersection-over-union (IoU) between the predicted bounding box and the ground-truth bounding box. The success plot is commonly used to visualize the percentage of frames where IoU exceeds a predetermined threshold, and the success rate is calculated as the area under the curve ($AUC$) of the success plot. Precision (P) is defined as the Euclidean distance between the predicted center point of the bounding box and the ground-truth center point. Normalized precision ($P_{\mathrm{norm}}$) is computed by scaling the accuracy with respect to the size of the ground-truth object.

Experiments on UAV123 datasets. The UAV123 dataset comprises 123 video sequences that are captured by low-altitude unmanned aerial vehicles (UAVs). The average sequence length is 915 frames. Based on the results presented in

Table 2. Comparison on LaSOT [46] and UAV123 [50] datasets. The best two results are shown in red and blue type.

Method	Epochs	Structure	UAV123		LaSOT
			$\mathit{AUC}$	$\mathit{P}$	$\mathit{AUC}$	${\mathit{P}}_{\mathit{Norm}}$	$\mathit{P}$
ParallelTracker	240	Transformer	69.3	90.5	68.8	78.4	74.9
OSTrack [18]	300	Transformer	68.3	88.8	69.1	78.7	75.2
STARK [19]	500	Transformer	68.2	89.1	67.1	77.0	66.1
TCTrack [36]	100	Transformer	60.4	80.0	-	-	-
DualTFR [38]	300	Transformer	68.2	-	63.5	72.0	66.5
DTT [51]	400	CNN+Transformer	-	60.1	-	-
TREG [52]	50	CNN+Transformer	66.9	88.4	64.0	74.1	-
TransT [32]	1000	CNN+Transformer	68.1	-	64.9	73.8	69.0
TrDimp [33]	50	CNN+Transformer	67.5	-	63.9	-	61.4
SAOT [53]	200	CNN	-	61.6	70.8	-
AutoMatch [54]	25	CNN	-	58.2	-	59.9
STMTracker [55]	20	CNN	64.7	-	60.6	69.3	63.3
SiamRCNN [11]	2720	CNN	64.9	83.4	64.8	72.2	-
PrDimp [12]	50	CNN	68.0	-	59.8	68.8	60.8
Ocean [13]	50	CNN	-	-	56.0	65.1	56.6
FCOT [14]	105	CNN	65.6	87.3	57.2	67.8	-
SiamGAT [15]	20	CNN	64.6	84.3	53.9	63.3	53.0
GGACD [16]	20	CNN	63.3	83.3	51.8	62.6	-
SiamFC++[17]	20	CNN	-	-	54.4	62.3	54.7
DiMP [29]	50	CNN	65.4	-	56.9	65.0	56.7

, our tracker outperforms all the other trackers. Our tracker exhibits a significantly faster convergence speed, requiring fewer training epochs while maintaining high performance, as compared to the state-of-the-art transformer-based models OStrack [18] and STARK [19]. When using 60 k image pairs per epoch, our tracker reduces the required training epochs by approximately 20% compared to OStrack. Additionally, our method achieves convergence twice as fast as STARK. These findings suggest that our approach has the potential to accelerate the convergence speed of tracking algorithms. It is worth noting that, despite using only 100 epochs, TCTrack [36] does not converge, which resulted in a worse performance.

ParallelTracker achieves a top-ranked $AUC$ score of 69.3%, surpassing the second-best OSTrack by 1.0% and third-best STARK by 1.1%; the latter two represent the most advanced level of object tracking. Furthermore, ParallelTracker achieves the top-ranked precision (P) of 90.46%, surpassing OSTrack by 1.7% and STARK by 1.4%. In the various scenarios of the UAV123 dataset, ParallelTracker consistently achieves good results.

For instance, as shown in Figure 4 and Figure 5, ParallelTracker demonstrates a statistically significant improvement over the second-best STARK by 1.1% in terms of $AUC$ and 6.7% P in the “full occlusion” scenario and surpasses the second-best OStrack by 1.7% $AUC$ and 3.0% P in the “partial occlusion” scenario. These results indicate that our algorithm is more robust to occlusions in images captured by unmanned aerial vehicles than other state-of-the-art trackers.

In the “scale variation”scenario, as shown in Figure 6, our tracker outperforms the second-best tracker, STARK, by 1.1% in terms of $AUC$ and 1.6% in terms of precision (P).

This finding illustrates that our tracker can effectively leverage prior knowledge of the tracked object’s scale invariance, thereby mitigating the impact of scale variations on its features. Moreover, in scenarios of “background clutter” and “low resolution” (as illustrated in Figure 7 and Figure 8), our tracking algorithm outperforms the second-best algorithm by 1.5% in $AUC$ and 4.85% in precision (P) for the former, and by 1.8% in $AUC$ and 3.3% in P for the latter. These results demonstrate the algorithm’s capacity to accurately extract discriminative object information amidst background disturbances and noise.

Figure 9 and Figure 10 demonstrate that our tracker outperforms the second-best tracker STARK by 1.4% in terms of $AUC$ and 2.1% in terms of P in the “out-of-view” scenario, and 1.3% in terms of $AUC$ and 1.6% in terms of P in the “viewpoint change” scenario. These results validate the effectiveness of our template update mechanism and its capability to address tracking drift more efficiently than other trackers.

Experiments on LaSOT datasets. The LaSOT test set comprises 280 long sequence videos captured at close range. We use this set to evaluate the long-term tracking performance of our ParallelTracker in close-range scenarios. As shown in Table 2, our tracker achieves a comparable level of performance to the best OSTracker in long-term tracking and outperforms all the other methods. ParallelTracker achieves a performance of 68.5% on the AUC metric, which is only 0.04% lower than that of OSTrack. Additionally, ParallelTracker surpasses the second-ranked STARK by 2.4%. The success plots and precision plots can be found in Figure 11. The experiments demonstrate our method not only is superior for UAV videos, but also can be well generalized to close-range scenes.

Visualization. We visualize the tracking boxes during the tracking process in Figure 12, demonstrating that our tracker is robust in scenarios such as tracking small objects, dealing with occlusions, and handling fast movements.

4.3. Ablation Study and Analysis

To verify the effectiveness of the newly designed components of ParallelTracker, we conduct ablation experiments on the UAV123 dataset.

4.3.1. Study on PEM

The PEM module is a critical component in the first block of ParallelTracker, intended to extract spatial prior information from image content, enhancing the convergence speed of model training and improving inference accuracy during testing. To verify our design, we conduct several experiments: (1) we remove the PEM module; (2) we utilize the PEM module to extract spatial prior information from each image separately; (3) we extract spatial prior information from all images. The results are presented in

Table 3. Ablation for PEM.

#	Module	Epochs	Params	FLOPs	AUC
1	No PEM	500	21.42 M	20.95 G	63.95
2	PEM on search region	240	27.35 M	26.80 G	65.24
3	PEM on online-template	240	26.76 M	26.03 G	66.42
4	PEM on template	240	26.76 M	26.03 G	69.29
5	PEM on all images	240	47.57 M	47.14 G	64.82

.

As shown in Table 3, we observe a significant drop in accuracy (approximately 5.28%) and a slower convergence speed when we remove the PEM module. The epochs required for model convergence are about 2.5 times longer compared to using PEM. By comparing experiments #2 and #3 with experiment #4, extracting spatial prior information from the template generated by the target initial frame effectively improves the AUC of our tracker on the UAV123 dataset by approximately 4.05%. We compare experiments #2, #3, and #4 with #5, and find that incorporating the spatial prior information of all images as learnable tokens and inputting them into the tracker does not improve its performance. Instead, it increases the number of parameters while decreasing the tracking accuracy. These findings demonstrate that incorporating key spatial prior knowledge can effectively reduce the training difficulty of the tracker and improve tracking accuracy.

4.3.2. Study on TPM

The TPM module is the second core component in ParallelTracker. The goal is to leverage prior knowledge to enhance the image features of the template. The results are shown in

Table 4. Ablation for TPM.

	TPM	FPS	AUC
ParallelTracker	-	29	65.71
ParallelTracker	✓	25	69.29

.

Incorporating the TPM module enables our tracker to increase the AUC by 3.58% on the AUV123 dataset, but it also decreases the FPS during the tracking process. These results demonstrate the importance of the TPM module in constructing a high-precision tracker.

4.3.3. Study on TPKM

The TPKM module is the final module in the ParallelTracker architecture. The objective is to leverage position attention and channel attention to facilitate information interaction from the fused sequence features, resulting in sequence features with image content correlation, which can be fed into the subsequent processing module. Our ablation experiments comprise the following components: (1) leveraging only location attention, (2) leveraging only channel attention, and (3) leveraging both types of attention in parallel. The results presented in

Table 5. Ablation for TPKM.

	Position Attention	Channel Attention	AUC
ParallelTracker	✓	-	69.12
ParallelTracker	-	✓	69.04
ParallelTracker	✓	✓	69.29

demonstrate that our designed position- and channel-parallel attention mechanism achieves an AUC of 69.29% on the UAV123 dataset, which is superior to either of the individual attention mechanisms.

4.3.4. Study on PSH

As presented in

Table 6. Ablation for PSH.

	Online	PSH	AUC
ParallelTracker	-	-	68.42
ParallelTracker	✓	-	67.57
ParallelTracker	✓	✓	69.29

, solely utilizing the online-update ParallelTracker that updates at a fixed interval led to inferior performance. Our online ParallelTracker, which incorporates our designed PSH module, achieves the highest AUC score among all methods evaluated on the UAV123 dataset. This finding highlights the crucial role of the PSH module in selecting a reliable online template.

4.3.5. Sensitivity Analysis for ${\lambda }_{L_1}$ and ${\lambda }_{GIoU}$

We leverage the lambda parameter setting in DETR [41], as shown in Equation (4). To evaluate the effect of different parameter settings in our tracker, we conduct sensitivity analysis experiments on the UAV123 dataset, as detailed in

Table 7. Sensitivity analysis for ${\lambda }_{L_1}$ and ${\lambda }_{GIoU}$.

	${\mathit{\lambda }}_{{\mathit{L}}_1}$	${\mathit{\lambda }}_{\mathit{GIoU}}$	AUC
ParallelTracker	5	2	68.42
ParallelTracker	2	5	68.27
ParallelTracker	1	1	68.38

. Because Equation (4) is only active during Stage 1, we employ this stage of the ParallelTracker for testing. It can be concluded that different weights of the two parameters have little impact on the accuracy.

5. Conclusions

We have developed a new end-to-end tracking algorithm, called ParallelTracker, which incorporates prior knowledge and parallel attention mechanisms to integrate image priors with feature extraction and interaction processes. Specifically, the PEM module is designed to address the lack of spatial prior information in the VIT-based tracker and enhance efficiency, the TPM module leverages the prior information in the template image to extract object-oriented and discriminative features in searched frames, and the TPKM module facilitates information exchange between templates and search regions. In addition, PSH incorporates an object template update strategy to enhance the tracker’s ability to accommodate changes in object shape and occlusions.

Experimental results demonstrate that ParallelTracker outperforms state-of-the-art algorithms in UAV videos. ParallelTracker can also maintain high levels of accuracy comparable to the latest methods in close-range video tracking scenarios, showing its powerful generalizing ability. Moreover, ParallelTracker can significantly reduce the epochs required by the popular methods to achieve convergence. This leads to a noteworthy reduction in the convergence difficulty of the VIT-based tracker.