Robust Correlation Tracking for UAV with Feature Integration and Response Map Enhancement

Abstract

Recently, correlation filter (CF)-based tracking algorithms have attained extensive interest in the field of unmanned aerial vehicle (UAV) tracking. Nonetheless, existing trackers still struggle with selecting suitable features and alleviating the model drift issue for online UAV tracking. In this paper, a robust CF-based tracker with feature integration and response map enhancement is proposed. Concretely, we develop a novel feature integration method that comprehensively describes the target by leveraging auxiliary gradient information extracted from the binary representation. Subsequently, the integrated features are utilized to learn a background-aware correlation filter (BACF) for generating a response map that implies the target location. To mitigate the risk of model drift, we introduce saliency awareness in the BACF framework and further propose an adaptive response fusion strategy to enhance the discriminating capability of the response map. Moreover, a dynamic model update mechanism is designed to prevent filter contamination and maintain tracking stability. Experiments on three public benchmarks verify that the proposed tracker outperforms several state-of-the-art algorithms and achieves a real-time tracking speed, which can be applied in UAV tracking scenarios efficiently.

1. Introduction

Visual object tracking has emerged to be a research hotspot in machine vision, as it is widely adopted in various fields, such as pedestrian tracking, traffic control, automatic navigation, human-machine interaction, medical imaging, and so on [1]. The purpose of general tracking is to automatically estimate the location of a moving object in subsequent frames based on its initial information given by the first frame. With the continuous development of commercial unmanned aerial vehicles (UAVs), corresponding practical applications involving aerial tracking [2,3,4], bathymetric monitor [5,6], aerial photogrammetry [7], disaster response [8] and petrographic investigation [9] have increased rapidly. However, the integration of object tracking techniques into UAV systems remains a challenging issue due to many visual interference factors and specific application constraints [10], such as inadequate sampling resolution, fast motion issue, severe visual occlusion, acute illumination variation, visual angle change, and scarce computation resources [11].

Traditional tracking algorithms are commonly divided into two types, i.e., generative methods [12,13,14] and discriminative methods [15,16,17,18]. Generally, generative methods focus on seeking a compact object representation of the object’s appearance while ignoring background information, which leads to severe performance degradation in complex tracking scenarios. In contrast, discriminative methods employ information from both the target and background to build the object appearance model. Discriminative trackers, which consider the object tracking problem as a binary classification task, are popular for their high accuracy and robustness. Recently, trackers based on correlation filters (CFs) [19,20,21,22,23,24,25] and deep learning (DL) technology [26,27,28,29,30,31,32] theory have emerged and developed rapidly. DL-based trackers leverage deep feature representations and large-scale offline learning to achieve state-of-the-art performance. The hierarchical convolutional features (HCF) [19] tracker takes advantage of deep features from three convolutional layers of the pre-trained network. The keyfilter-aware object tracker (KAOT) [20] also utilizes deep convolutional features to enhance the robustness of UAV tracking. The fully-convolutional Siamese network (SiamFC) [21] algorithm employs an end-to-end architecture to learn a similarity measure function offline, and detects the object that is most similar to the template online. Motivated by the success of SiamFC, some further improvements have been made, such as combining the Siamese network with a region proposal network (RPN) [22,33], adding additional branches [23], exploiting the Transformer architecture [24,34], and so on. In addition, the Siamese network-based algorithm with an anchor proposal network (APN) [25] module is developed for onboard aerial tracking. However, DL-based trackers require vast amounts of data and high-end graphics processing units (GPUs) for training deep neural networks. It is worth noting that most UAVs are currently not equipped with GPUs due to considerations of cost per unit and the weight of the drone itself. Therefore, deploying deep learning models is not an ideal choice for visual object tracking tasks in aerial videos.

Despite performing slightly inferior to DL-based trackers, CF-based trackers can achieve better balance among tracking precision and running speed, and the low cost, correlation filter has become a prominent paradigm for resource-limited UAV platforms. The minimum output sum of the squared errors (MOSSE) [26] tracker is a pioneering work in the development of the CF paradigm. Inspired by MOSSE, the kernelized correlation filter (KCF) [27] algorithm introduces the circulant matrix and kernel trick to CF-based tracking. The scale adaptive with the multiple features (SAMF) [28] tracker integrates the raw pixels, histogram of oriented gradient (HOG) [35], and color names (CN) [29] features to represent the target object. Somewhat differently, the online multi-feature learning (OMFL) [30] algorithm for UAV tracking utilizes the peak-to-sidelobe ratio (PSR) to weight and fuses the response maps learned from different types of features. Recently, the interval-based response inconsistency (IBRI) [31] and the future-aware correlation filter (FACF) [32] trackers handle the model drift issue in aerial tracking scenarios using historical information and future context information, respectively. The CF paradigm has also been extended by saliency proposals [36], multi-cue fusion [37], kernel cross-correlator [38], particle filter [39] and rotation prediction [40].

In CF-based tracking, a correlation filter model is trained with the ability to classify the compact samples by minimizing a least-squares loss. Then, in consecutive frames, the target is detected on the region of interest (ROI) by applying the trained filter. There are two core characteristics of this tracking-by-detection paradigm. First, massive virtual samples can be obtained by the cyclic shift operation of the ROI. Hence, CF-based trackers have sufficient samples for online training. Second, correlation in the spatial domain can be calculated by the discrete Fourier transform (DFT) as a multiplication of elements in the frequency domain. Such an advantage greatly accelerated the training and detection procedures of CF-based trackers, and thus real-time tracking is realized. However, learning from cyclicly shifted examples in the classical CF paradigm [27] is not always valid. Specifically, since the training patches contain synthetic negative samples, periodic repetitions in such synthetic samples introduce disturbing boundary effects that may deteriorate the discriminative capability of the filters. In previous studies, several approaches have been proposed aiming at reducing the boundary effects while offering adequate training patches [41,42,43,44]. Among these methods, a tracker known as a correlation filter with limited boundaries (CFLB) [41] tracker is proposed to extract real negative training samples by incorporating a binary matrix used for cropping patches from background regions. As an extended version of CFLB, the background-aware correlation filter (BACF) [43] employs the HOG [35] instead of the raw pixels in feature extraction phase. Although the BACF tracker solves the boundary effect problem, it still struggles to adapt to various challenging scenarios with a single type of feature to model the targets tracked by the UAV. Moreover, the BACF cannot resolve the model drift problem caused by occlusions or distractors, which inhibits the tracking robustness for its further development.

In this paper, we propose a robust BACF-based algorithm with feature integration and response map enhancement for the UAV tracking applications. The main contributions of this work are three-fold:

• A new feature integration method is presented to combine distinct gradient features in the HSV color space and their binary representation at a hierarchical level for building a discriminative appearance model of the aerial tracking object. This feature-level fusion manner not only makes use of the gradient and color information together in a more rational approach but also delivers better anti-interference ability with the benefit of binary representation.
• Saliency awareness is introduced into the correlation filter framework, enabling the tracker to focus on more salient object regions to enhance model discrimination and mitigate model drift. Furthermore, based on a consistent criterion that reveals the quality of the response map, an adaptive response fusion strategy and a dynamic model update mechanism are designed, respectively, for achieving a more reliable decision-level fusion result and avoiding model corruption.
• Exhaustive evaluations are performed on four popular UAV tracking benchmarks, i.e., UAV123@10fps [45], VisDrone2018-SOT [46], UAVTrack112 [25], and UAV20L [45]. The experimental results demonstrate that our approach achieves very competitive performance compared to state-of-the-art trackers, while delivering a real-time tracking speed of 26.7 frames per second (FPS) on a single CPU.

The rest of this paper is organised below: Section 2 details the tracking framework of the proposed method. Extensive experiments are conducted in Section 3. Section 4 discusses the results and analyzes the effectiveness of each component. At last, a brief conclusion of this study is given in Section 5.

2. Proposed Approach

Based on the CF paradigm, a novel tracking pipeline exploiting feature integration and response map enhancement is proposed. In Section 2.1, we first illustrate the proposed feature integration method. In Section 2.2, we describe the filter learning and detection stages of the baseline framework. In Section 2.3, we explain how saliency is incorporated into the framework and provide a description of our adaptive response fusion strategy. At last, the dynamic model update mechanism and the scale estimation method are illustrated in Section 2.4. The diagram in Figure 1 outlines the workflow of our tracker.

2.1. Feature Integration Method

Feature extraction can be viewed as one of the most crucial components in the tracking framework. Using adequate features can dramatically boost the tracking performance [47]. Previous works [27,48,49] have verified that gradient and color information is very effective for CF-based trackers. Moreover, there is a favorable complementary between the gradient and color features, since the former has a good tolerance to local geometric changes, while the latter possesses a certain amount of photometric invariance. Although exploiting more powerful features, such as those extracted from convolutional neural networks (CNNs) can yield state-of-the-art performance [50,51,52], it will incur a high computational cost, which is highly undesirable for online UAV tracking. Therefore, based on the handcrafted features, we develop a feature integration method to build a robust appearance model of the tracked object.

Inspired by the work in [36], the input image is converted firstly from RGB to the HSV (Hue, Saturation, and Value) color space that corresponds more closely to the manner in which humans perceive color. The elements of each channel’s colormap are in the range of 0 to 1. Secondly, the HSV descriptor is converted to a binary representation by exploiting a simple threshold $\mu$. If the value is larger than the threshold $\mu$, the corresponding feature is set to 1, otherwise, it is set to 0. Compared with the standard HSV descriptor, the binary representation is insensitive to object appearance variations in local details and background interference, which facilitates tracking algorithms to adapt to complex UAV tracking scenarios. Further, our binary representation is extended to construct a group of HOG features [35] across each channel. It should be noted that only integrating the HOG features extracted from the binary representation is not sufficient to achieve precise localization for inadequate fine-grained descriptions; thus, these features are employed as auxiliary features in our approach. Instead, the HOG feature maps extracted from the input grayscale image (equivalent to the Value channel, V-channel for short) can be regarded as the primary features. All the primary and auxiliary components are finally concatenated to constitute our integrated features. The process of feature integration is displayed in Figure 2. Thereby, with the contribution of the binary representation, a more natural fusion of color and gradient information is achieved for UAV tracking.

2.2. Filter Learning and Detection

In this work, we utilize BACF [43] as our baseline tracker due to its effectiveness in reducing the boundary effects, as well as its real-time performance. As previously mentioned, our integrated features are exploited to enhance the robustness of the baseline tracker. More specifically, let $\mathrm{x}$ denote a training sample (image patch) with a large search region; T represents the number of training samples, $\mathrm{y}$ stands for the desired correlation response (predefined Gaussian-shaped labels), and the correlation filter $\mathrm{h}$ of the baseline tracker is learned by minimizing the objective function as follows:

$$E\left(\mathrm{h}\right)=\frac{1}{2}\sum _{j=1}^T\parallel {\mathrm{y}}_j-\sum _{k=1}^K{\mathrm{h}}_k^{\top }\mathbf{P}{\mathrm{x}}_k[\Delta {\tau }_j]{\displaystyle |}{{\displaystyle |}}_2^2+\frac{\lambda }{2}\sum _{k=1}^K\parallel {\mathrm{h}}_k{\parallel }_2^2,$$

(1)

where ${\mathrm{h}}_k$ and ${\mathrm{x}}_k$, respectively, denote the k-th channel of the learned filter and our vectorized fused feature, K denotes the number of feature channels, ${\mathrm{y}}_j$ indicates the j-th element of $\mathrm{y}$, $[\Delta {\tau }_j]$ stands for the circular shift operator, and ${\mathrm{x}}_k[\Delta {\tau }_j]$ employs a j-step circular shift to ${\mathrm{x}}_k$. $\mathbf{P}$ denotes a $D\times T$ binary matrix that crops the middle D elements of ${\mathrm{x}}_k$, and $\mathbf{P}{\mathrm{x}}_k[\Delta {\tau }_j]$ returns all shifted patches with the size of D from the original image patch. It is noted that the matrix $\mathbf{P}$ plays an important role in the baseline framework, which makes the correlation operator apply on the real negative samples. $\lambda$ represents a regularization, and the superscript ⊤ is the conjugate transpose of a matrix. In this formulation, ${\mathrm{x}}_k\in {\mathbb{R}}^T$, ${\mathrm{y}}_j\in {\mathbb{R}}^T$, and ${\mathrm{h}}_k\in {\mathbb{R}}^D$, where $T\gg D$. For computational efficiency, Equation (1) is rewritten in the frequency domain as follows:

$$\begin{array}{cc}\hfill E\left(\mathrm{h},\widehat{\mathrm{g}}\right)& =\frac{1}{2}\parallel \widehat{\mathrm{y}}-\widehat{\mathbf{X}}\widehat{\mathrm{g}}{\parallel }_2^2+\frac{\lambda }{2}{\parallel \mathrm{h}\parallel }_2^2\hfill \\ \hfill \mathrm{s}.\mathrm{t}.\widehat{\mathrm{g}}& =\sqrt{T}(\mathbf{F}{\mathbf{P}}^{\top }\otimes {\mathbf{I}}_K)\mathrm{h},\hfill \end{array}$$

(2)

where $\mathrm{h}=[{\mathrm{h}}_1^{\top },\cdots ,{\mathrm{h}}_K^{\top }]$, $\widehat{\mathrm{g}}$ denotes an auxiliary variable and $\widehat{\mathrm{g}}={[{\widehat{\mathrm{g}}}_1^{\top },\cdots ,{\widehat{\mathrm{g}}}_K^{\top }]}^{\top }$, the matrix $\widehat{\mathbf{X}}={\left[{diag(\widehat{\mathrm{x}}}_1\right)}^{\top },\cdots ,{diag(\widehat{\mathrm{x}}}_K{{)}^{\top }]}^{\top }$, the superscript $\widehat{A}$ represents the DFT operator of a signal A. $\mathbf{F}$ denotes the orthonormal $T\times T$ matrix for mapping a vectorized signal with T dimensions to the Fourier domain, ${\mathbf{I}}_K$ denotes a $K\times K$ identity matrix, and ⊗ stands for the Kronecker product. Employing the Augmented Lagrangian Method (ALM) and the alternating direction method of multiplier (ADMM) algorithm [53], Equation (2) can be solved by transformed into two subproblems, respectively, the following ${\mathrm{h}}^{*}$ and ${\widehat{\mathrm{g}}}^{*}$:

$$\left\{\begin{array}{cc}\hfill {\mathrm{h}}^{*}& =arg\underset{\mathrm{h}}{min}\left\{\frac{\lambda }{2}{\parallel \mathrm{h}\parallel }_2^2+{\widehat{\zeta }}^{\top }(\widehat{\mathrm{g}}-\sqrt{T}(\mathbf{F}{\mathbf{P}}^{\top }\otimes {\mathbf{I}}_K)\mathrm{h})\right.\hfill \\ \hfill & \left.+\frac{\sigma }{2}\parallel \sqrt{T}(\mathbf{F}{\mathbf{P}}^{\top }\otimes {\mathbf{I}}_K){\mathrm{h})\parallel }_2^2\right\},\hfill \\ \hfill {\widehat{\mathrm{g}}}^{*}& =arg\underset{\widehat{\mathrm{g}}}{min}\left\{\frac{1}{2}\parallel \widehat{\mathrm{y}}-\widehat{\mathbf{X}}\widehat{\mathrm{g}}{\parallel }_2^2+{\widehat{\zeta }}^{\top }(\widehat{\mathrm{g}}-\sqrt{T}(\mathbf{F}{\mathbf{P}}^{\top }\otimes {\mathbf{I}}_K)\mathrm{h})\right.\hfill \\ \hfill & \left.+\frac{\sigma }{2}\parallel \sqrt{T}(\mathbf{F}{\mathbf{P}}^{\top }\otimes {\mathbf{I}}_K){\mathrm{h})\parallel }_2^2\right\},\hfill \end{array}\right.$$

(3)

where $\sigma$ represents the penalty factor and ${\widehat{\zeta }}^{\top }={[{\widehat{\zeta }}_1^{\top },\cdots ,{\widehat{\zeta }}_K^{\top }]}^{\top }$. Each of the above subproblems can be resolved iteratively by the ADMM algorithm.

In the detection phase, the correlation response is calculated by applying the learned filter $\widehat{\mathrm{g}}$ as:

$${\mathbf{R}}_{\mathrm{i}}^{\left(t\right)}={\mathcal{F}}^{-1}(\sum _{k=1}^K{\widehat{\mathrm{x}}}_k^{\left(t\right)}\odot {\widehat{\mathrm{g}}}_k^{(t-1)}),$$

(4)

where ${\mathbf{R}}_{\mathrm{i}}^{\left(t\right)}$ is the correlation response map in frame t, the subscript $\mathrm{i}$ indicates that the response map corresponds to the integrated features (${\mathbf{R}}_{\mathrm{i}}$ is called the integrated feature response for short in the following parts), ${\mathcal{F}}^{-1}$ and ⊙ represent the inverse Fourier transform (IFT) operator and the element-wise product, respectively, ${\widehat{\mathrm{x}}}_k^{\left(t\right)}$ stands for the Fourier form of the extracted feature maps in frame t, and ${\widehat{\mathrm{g}}}_k^{(t-1)}$ represents the filter in the previous frame.

2.3. Tracking Incorporating Saliency

Visual saliency is widely deployed in object detection [54,55] and semantic segmentation [56,57]. We assume that it may be also capable of helping the CF-based tracking algorithm to distinguish the target from its background as a supplementary feature that provides object description from another perspective. Therefore, object saliency information is incorporated to enhance correlation tracking in our framework. Starting from the input image patch $\mathrm{x}$, a simple yet effective saliency detection algorithm [58] based on the spectral residual theory is applied in this patch to efficiently generate the saliency map. Specifically, we first construct the amplitude feature $\mathcal{A}\left(\mathrm{x}\right)$ and the phase feature $\mathcal{P}\left(\mathrm{x}\right)$ in the frequency domain through DFT calculation. Then, the log spectrum $\mathcal{L}\left(\mathrm{x}\right)=log\left(\mathcal{A}\right(\mathrm{x}\left)\right)$ is adopted to achieve the spectral residual $\mathcal{R}\left(\mathrm{x}\right)$ by Equation (5):

$$\mathcal{R}\left(\mathrm{x}\right)=\mathcal{L}\left(\mathrm{x}\right)-{\mathbf{H}}_n\star \mathcal{L}\left(\mathrm{x}\right),$$

(5)

where ⋆ stands for the convolution operator, and ${\mathbf{H}}_n$ represents the $n\times n$ matrix, expressed as:

$${\mathbf{H}}_n=\frac{1}{n^2}\left[\begin{array}{cccc}1& 1& \cdots & 1\\ 1& 1& \cdots & 1\\ ⋮& ⋮& \ddots & ⋮\\ 1& 1& \cdots & 1\end{array}\right].$$

(6)

The spectral residual contains the distinct information of an image. It captures a compressed representation of the noticeable objects. Through IFT calculation, the saliency map $\mathcal{S}\left(\mathrm{x}\right)$ can be generated in the spatial domain by Equation (7):

$$\mathcal{S}\left(\mathrm{x}\right)=\mathbf{G}\star {\mathcal{F}}^{-1}\left\{\exp \left(\mathcal{R}\right(\mathrm{x})+\mathcal{P}(\mathrm{x}\left)\right)\right\},$$

(7)

where $\mathbf{G}$ represents a Gaussian matrix used to smooth the saliency map.

Considering the saliency map as a kind of ordinary single-channel feature map, we train a specific correlation filter for saliency and further achieve the saliency response ${\mathbf{R}}_{\mathrm{s}}$, as described in Section 2.2. Illustration of the object saliency maps and their correlation response maps in our scheme is shown in Figure 3. It can be observed that embedding saliency into the BACF [43] framework is effective in highlighting the object’s corresponding region and suppressing background interference in the response map. The saliency awareness works as a spatial attention that reflects the foreground information to assist the integrated feature response ${\mathbf{R}}_{\mathrm{i}}$ and thus avoid the tracker drifting to the background.

In the response fusion stage, the original saliency response is first interpolated and adjusted to the same size as the integrated feature response output. Subsequently, assuming that we directly perform response maps’ integration, the variant interference and the coarse-grained property of saliency will cause the fusion results to inevitably contain unwanted noise. Therefore, an adaptive response fusion strategy based on the confidence of squared response map (CSRM) [59] is introduced to weigh and merge the two response maps in this work. CSRM reflects the degree of fluctuation and the confidence level of the response maps. For a response map $\mathbf{R}$, which is a $M\ast N$ matrix, the CSRM score $\omega$ is defined as:

$$\omega =\frac{{\left|{\max \left(\mathbf{R}\right)}^2-{\min \left(\mathbf{R}\right)}^2\right|}^2}{\frac{1}{MN}{\displaystyle \sum _{m=1}^M}{\displaystyle \sum _{n=1}^N}{\left|{\mathbf{R}}_{mn}^2-{\min \left(\mathbf{R}\right)}^2\right|}^2}.$$

(8)

The numerator and the denominator of the CSRM represent the peak and the mean square value of the response map, respectively. We use Equation (8) to compute the CSRM scores ${\omega }_{\mathrm{i}}$ and ${\omega }_{\mathrm{s}}$ for the response maps ${\mathbf{R}}_{\mathrm{i}}$ and ${\mathbf{R}}_{\mathrm{s}}$. A higher CSRM score means better quality of the response map. Hence, we express the weight of the integrated feature response ${\mathbf{R}}_{\mathrm{i}}$ as the percentage of its CSRM score ${\omega }_{\mathrm{i}}$ in the total score obtained by adding ${\omega }_{\mathrm{i}}$ and ${\omega }_{\mathrm{s}}$, i.e., $\frac{{\omega }_{\mathrm{i}}}{{\omega }_{\mathrm{i}}+{\omega }_{\mathrm{s}}}$. The weight of the saliency response ${\mathbf{R}}_{\mathrm{s}}$ can be determined in the same way. Then, the response maps are linearly added using the adaptive weights to obtain the enhanced response map ${\mathbf{R}}_{\mathrm{e}}$, as follows:

$${\mathbf{R}}_{\mathrm{e}}=\frac{{\omega }_{\mathrm{i}}}{{\omega }_{\mathrm{i}}+{\omega }_{\mathrm{s}}}{\mathbf{R}}_{\mathrm{i}}+\frac{{\omega }_{\mathrm{s}}}{{\omega }_{\mathrm{i}}+{\omega }_{\mathrm{s}}}{\mathbf{R}}_{\mathrm{s}}.$$

(9)

In the enhanced response map, the coordinate location with the maximum value indicates the estimated target position.

2.4. Model Update and Scale Estimation

Due to camera movements and variations in the viewpoint, the target appearance often changes notably in UAV tracking scenes. Consequently, the learned model is expected to adapt to such variations depending on an online model update mechanism. Nevertheless, when the target experiences more various challenging scenarios such as occlusion and background clutter, unreliable tracking results and inopportune model update operations will contaminate the filter model over time, which can also lead to model drift if not immediately addressed. In this study, we design the dynamic model update mechanism based on a simple fact that the response map appears as a sharp peak and the corresponding CSRM score is higher if the object is not occluded by the surroundings. On the contrary, the response map may appear as a relatively flat peak or even multiple peaks and the corresponding CSRM score is lower if the target is experiencing occlusion. Therefore, we introduce a threshold $\theta$ to distinguish whether the tracking result obtained by the enhanced response map ${\omega }_{\mathrm{e}}$ is reliable or not. To eliminate variance effects among different UAV videos, we use normalized CSRM scores for comparison with the threshold. An illustration of normalized CSRM score and response map variations with and without occlusion is shown in Figure 4. For the current frame t, if the tracking result is reliable, it can be formulated as:

$$\frac{{\omega }_{\mathrm{e}}^{\left(t\right)}}{{\omega }_{\mathrm{e}}^{\left(2\right)}}>\theta ,$$

(10)

where $\frac{{\omega }_{\mathrm{e}}^{\left(t\right)}}{{\omega }_{\mathrm{e}}^{\left(2\right)}}$ denotes the normalized CSRM score, ${\omega }_{\mathrm{e}}^{\left(2\right)}$ is the first CSRM score of the enhanced response map (calulated at the second frame) in a video sequence.

When Equation (10) is satisfied, the object template of the current frame is considered reliable. Then, we independently update the integrated feature model to accommodate the object appearance changes, as follows:

$${{\widehat{\mathrm{x}}}_{\mathrm{model}}^{\left(t\right)}}^{}=(1-\alpha ){{\widehat{\mathrm{x}}}_{\mathrm{model}}^{(t-1)}}^{}+\alpha {{\widehat{\mathrm{x}}}_{\mathrm{model}}^{\left(t\right)}}^{},$$

(11)

where ${{\widehat{\mathrm{x}}}_{\mathrm{model}}^{\left(t\right)}}^{}$ and ${{\widehat{\mathrm{x}}}_{\mathrm{model}}^{(t-1)}}^{}$, respectively, denote the current frame model and the previous frame model, and $\alpha$ denotes the learning rate. In case the current tracking result (or, in other words: current object template) is unreliable, the integrated feature model is retained without updating it to prevent the degradation of the filter. Considering that the initial target information provides the most credible object template, the saliency model in our mechanism is initialized in the first frame and then remains unchanged for the subsequent frames.

For the scale estimation module, we remove the saliency information for algorithmic acceleration. The correlation filter with the integrated features is applied on five scale search regions, and thus five related response maps are obtained by correlation operations. The estimated target scale is determined by the scale factor corresponding to the maximum value in all of these response maps. To sum up, the main steps of the proposed approach are shown in Algorithm 1.

Algorithm 1: The proposed tracker.

3. Experiments

In this section, the proposed algorithm is compared with other 14 state-of-the-art CF-based tracking algorithms recently proposed, i.e., ECO-HC [48], BACF [43], CSR-DCF [44], CACF [60], STRCF [61], MCCT-H [37], KCC [38], LDES [40], ARCF [2], OMFL [30], AutoTrack [3], DR2Track [4], IBRI [31], and SITUP [62]. We employ two benchmark metrics [63] involving precision and success rate to access tracking accuracy for all tested trackers. In addition, the average FPS is deployed for tracking speed measurement.

3.1. Implementation Details

All compared trackers and ours are implemented in MATLAB R2017a on an Intel Core i7-8700 3.2 GHz CPU with 16 GB RAM. The main parameter settings of our tracker are presented as follows: the threshold $\mu$ for binary representation is chosen as 0.5, the regularization coefficient $\lambda$ is set to 0.001, the penalty factor $\sigma$ is set to 1, the threshold $\theta$ for determining the reliability of the enhanced response map is 0.45, and the learning rate $\alpha$ is set to 0.0127. All parameter values remain fixed in our experiments.

3.2. Quantitative Evaluation

3.2.1. Overall Performance Evaluation

To comprehensively evaluate the overall performance, we first conduct the experiments on three common UAV tracking benchmarks, i.e., UAV123@10fps [45] and Vis Drone2018-SOT [46], and UAVTrack112 [25]. Precision and success plots of our algorithm and other compared trackers on these three benchmarks are presented in Figure 5, Figure 6 and Figure 7. Compared to baseline tracker BACF [43], our approach has a significant advantage of 10.35% in precision and 6.84% in success rate. As shown in Figure 5, the proposed tracker achieves the best performance in both precision and success rate on the UAV123@10fps benchmark. As presented in Figure 6, our tracker consistently ranks first in both terms of precision and success rate on the VisDrone2018-SOT benchmark, which significantly owns an advantage of 1.39% and 1.47% over the second-best tracker. As shown in Figure 7, our tracker ranks first place in precision and second place in success rate on the UAVTrack112 benchmark. Subsequently, all trackers are also tested on a long-term benchmark UAV20L [45]. The evaluation results are presented in Figure 8. The results indicate that the proposed tracker realizes the best performance in success rate, and the second-best performance in precision, which is slightly lower than that of OMFL [30].

Besides satisfactory tracking accuracy, the tracking speed of our algorithm is capable of meeting the real-time requirements, as shown in

Table 1. Real-time performance comparison on the UAV123@10fps [45] benchmark. All results are generated solely by a desktop CPU.

Tracker	Average FPS	Real-Time	Venue
ECO-HC [48]	52.0	Yes	2017’CVPR
CSR-DCF [44]	14.2	No	2017’CVPR
BACF [43]	52.0	Yes	2017’ICCV
CACF [60]	26.2	Yes	2017’ICCV
STRCF [61]	25.9	Yes	2018’CVPR
MCCT-H [37]	57.6	Yes	2018’CVPR
KCC [38]	35.0	Yes	2018’AAAI
LDES [40]	7.0	No	2019’AAAI
ARCF [2]	28.1	Yes	2019’ICCV
OMFL [30]	14.4	No	2019’Remote Sensing
AutoTrack [3]	57.7	Yes	2020’CVPR
DR2Track [4]	44.0	Yes	2020’IROS
SITUP [62]	12.9	No	2020’TIP
IBRI [31]	25.7	Yes	2021’TGARS
Ours	26.7	Yes	This work

.

3.2.2. Attribute-Based Evaluation

To analyze the tracking performance of UAV-specific challenging aspects in more detail, the UAV123@10fps [45] benchmark annotates the video sequences into 12 different attributes, i.e., aspect ratio change (ARC), background clutter (BC), camera motion (CM), fast motion (FM), full occlusion (FOC), illumination variation (IV), low resolution (LR), out-of-view (OV), partial occlusion (POC), similar object (SOB), scale variation (SV), and viewpoint change (VC). We report the experimental results of attribute-based performance comparisons in Figure Figure 9 and Figure 10. The observations reveal that the proposed approach performs very competitively over other compared methods. In terms of precision, our approach ranks top among the 10 attributes involving ARC, CM, FM, FOC, IV, OV, POC, SOB, SV, and VC. As shown in Figure 9, our tracker performs 0.23%, 1.43%, 3.61%, 0.32%, 0.95%, 4.60%, 2.08%, 2.12%, 0.33%, and 3.21% better on these attributes compared to the second-best tracker. Note that the proposed algorithm also surpasses the baseline BACF [43] by a large margin (more than 10%) on these 10 attributes. In terms of success rate, Figure 10 shows that our approach obtains the best results on all attributes except BC, LR, and SV. Especially under the visual interference factors unique to UAV tracking, our tracker achieves notable improvements compared to the second-best tracker, i.e., 2.75% on CM, 2.85% on FM, 3.21% on IV, and 2.73% on VC.

3.3. Qualitative Evaluation

For a more intuitive evaluation, qualitative comparisons are conducted with the proposed tracker and other five representative trackers including ARCF [2], OFML [30], AutoTrack [3], IBRI [31], and the baseline method BACF [43]. Figure 11 illustrates some keyframes with the tracking results on six video sequences (from the UAV123@10fps [45] benchmark), i.e., Car1-2, Person19, Wakeboard5, Uav1-2, Car1-s, and Person3-s. In the Car1-2 sequence, the target with a small size is greatly disturbed by the similar objects around. It can be observed that BACF and OFML lose the target, while our tracker keeps tracking it successfully. In the Person19 sequence, the target moves out of view gradually. Notably, among all the trackers, only IBRI and ours can re-identify the target when it appears again. In the Wakeboard5 sequence, the target experiences significant deformation and scale variations. BACF, AutoTrack, and IBRI completely drift away from the target, while our tracker estimates both the position and scale accurately. In the Uav1-2 sequence, all the comparative trackers do not perform well due to the fast motion and viewpoint change. In contrast, our tracker effectively adapts to these two attributes. As for the Car1-s and Person3-s sequences which are generated by the UE4-based simulator [45], it should be noted that all the trackers except ours fail to locate the targets in the complex scenarios with multiple interference factors including poor illumination, severe viewpoint change, and scale variation.

4. Discussion

In general, the quantitative and qualitative experimental results demonstrate the effectiveness and robustness of the proposed algorithm in UAV tracking. As shown in Figure 5, Figure 6, Figure 7 and Figure 8, on each of the four public benchmarks [25,45,46], our tracker obtains the best or the second-best results in terms of precision and success rate. The overall evaluation verifies the superiority of our algorithm in short-term UAV tracking, as well as in long-term tracking scenarios when severe occlusion and out-of-view conditions occur. As shown in Figure 9 and Figure 10, our tracker performs well on most attributes and is applicable for almost all types of complex scenes. The attribute-based evaluation confirms the robustness and well-roundness of our proposed algorithm under different tracking challenges. Meanwhile, the qualitative results further validate the effectiveness of our algorithm and its capability of suppressing model drifts caused by occlusion, out-of-view, or background interference.

To analyze the impact of each component in our tracking architecture, we provide the performance of our tracker with different modules on the benchmark UAV123@10fps [45] in

Table 2. Ablation study of our tracker on the UAV123@10fps [45] benchmark.

Trackers	Precision	Success Rate	Averge FPS
Baseline (BACF) [43]	57.02%	41.02%	52.0
Baseline + FI	65.12%	46.25%	31.2
Baseline + FI + RME	66.23%	47.35%	24.1
Baseline + FI + RME + DMU (Final)	67.37%	47.86%	26.7

. The basic notions are as follows: FI denotes feature integration, RME stands for response map enhancement, and DMU means dynamic model update. Table 2 shows that all of the three modules contribute to the substantial improvement over the baseline tracker [43]. The integrated features boost the tracking performance in terms of both precision and success rate by a large margin. The enhanced response map achieved by incorporating the saliency response adaptively further improves the tracking accuracy. Our simple model update mechanism can, not only avoid model contamination, but also effectively reduce computational complexity.

5. Conclusions

In this work, a novel CF-based algorithm is proposed to implement UAV tracking tasks. By virtue of an effective feature integration method, the correlation filter with integrated features deliver a significant improvement in adapting to different challenges such as object geometry changes and illumination variations. Saliency awareness is then integrated into the tracking framework, which facilitates the mitigation of model drift. Afterwards, decision-level fusion is achieved through an adaptive response fusion strategy, resulting in an enhanced response map with relatively little background interference. Finally, we utilize a dynamic model update mechanism to maintain the model stability in the presence of occlusions. Extensive experiments on four UAV tracking benchmarks validate the superiority of the proposed tracker over other state-of-the-art trackers. Moreover, our tracker is capable of efficient tracking with low computational requirements, which is promising for further development of real-time UAV applications.