Learning Augmented Memory Joint Aberrance Repressed Correlation Filters for Visual Tracking

Abstract

With its outstanding performance and tracking speed, discriminative correlation filters (DCF) have gained much attention in visual object tracking, where time-consuming correlation operations can be efficiently computed utilizing the discrete Fourier transform (DFT) with symmetric properties. Nevertheless, the inherent issues of boundary effects and filter degradation, as well as occlusion and background clutter, degrade the tracking performance. In this work, we proposed an augmented memory joint aberrance repressed correlation filter (AMRCF) for visual tracking. Based on the background-aware correlation filter (BACF), we introduced adaptive spatial regularity to mitigate the boundary effect. Several historical views and the current view are exploited to train the model together as a way to reinforce the memory. Furthermore, aberrance repression regularization was introduced to suppress response anomalies due to occlusion and deformation, while adopting the dynamic updating strategy to reduce the impact of anomalies on the appearance model. Finally, extensive experimental results over four well-known tracking benchmarks indicate that the proposed AMRCF tracker achieved comparable tracking performance to most state-of-the-art (SOTA) trackers.

1. Introduction

Visual object tracking, as a fundamental task of computer vision and pattern recognition, is widely applied in biological vision, autonomous driving, video surveillance and other fields. The task of visual tracking is to predict the spatial position and scale size of an arbitrary target in a video or continuously associated image sequence given limited information (i.e., initial target position and scale information). Due to the variable tracking environment, the tracked object is easily suffered from various factors, such as illumination variation, deformation, partial/full occlusion, causing lost target and tracking failure. Therefore, achieving stable and accurate tracking of an object under complex environments is still a challenge.

Since Bolme [1] et al. first proposed the minimum output sum of the squared error (MOSSE) tracker, which achieved promising accuracy and very fast speed (615 FPS) only using the single-channel greyscale features, DCF-based tracking methods have become a hot topic in visual tracking. By means of DFT with conjugate symmetry, the time-consuming correlation operation can be efficiently solved in the frequency domain. Inspired by the MOSSE, CSK [2] creatively adopts circular shift instead of random sampling to increase the training samples and introduced kernel functions to improve the computational speed. While the CSK-based multi-channel kernel correlation filter (KCF) [3] achieved a great improvement in tracking performance, the follow-up trackers have made significant advancements over scale adaptation [4,5,6], feature representation and fusion scheme [7,8,9], models innovation [10,11,12], etc., however, several inherent problems remain. First, the training samples based on the periodic assumption are low-quality synthetic samples that produce unwanted boundary effects, making the filter’s discriminatory power less than optimal. Second, to accommodate the appearance changes, most trackers maintain and update an appearance model frame by frame with a pre-determined fixed learning rate. This ignores the appearance variations and lacks the integration of historical appearances information, leading to model degradation and reducing its effectiveness. Third, in the case of challenging scenarios such as partial/full occlusion and background clutter, the object appearance changes drastically while the output confident maps become distorted, i.e., response anomalies, which may cause the target drift and tracking failure.

To deal with the above challenges, the existing SOTA trackers mainly focus on two aspects to improve tracking performance: (a) by imposing spatial or temporal regularization constraints whilst constructing the model, and (b) by analysing the inherent information of the confidence map to construct feedback loops to optimize the entire tracking framework. Spatially regularized discriminative correlation filter (SRDCF) [13] introduces a spatial regularization that penalizes the filter coefficients at the boundaries, allowing more energy to be concentrated in the central region. CSR-DCF [14] distinguishes the foreground from the background by colour segmentation in the search area, while the foreground mask matrix is utilized to select the filter coefficients. A background-aware correlation filter (BACF) [15] utilizes a binary mask matrix to crop real samples from the search window and achieve the suppression of background information. However, it is not reasonable to impose fixed constraints on the filter, which does not reflect well the changing characteristics and appearance of the tracking target. Furthermore, by introducing temporal consistency constraints [16,17,18], the aim is that the model does not over-change during the tracking of the same target, thus mitigating the model’s degradation. Nevertheless, the existence of non-ideal factors such as occlusion and background interference make this assumption difficult to achieve. Since minimizing the impact of response anomalies, on the one hand, Huang et al. [19] directly introduced a regularization constraint via comparing the Euclidean distance between the current and the previous output response maps to limit the response distortion. Meanwhile, many works [20,21,22] have developed high-confidence updating strategies to construct feedback loops, which have effectively improved the tracking performance. However, these trackers hardly exploit the intrinsic information provided by different historical views, increasing the risk of model drift.

In this work, we proposed an augmented memory joint aberrance repressed correlation filter (AMRCF) tracking method to address the above limitation. Based on the excellent BACF tracker, we incorporate the adaptive spatial regularization constraint to mitigate the boundary effects. To mitigate the model degradation, several historical views are selected to train the filter model together with the current view, making the trained model adapt to the new target appearance and remember the previous ones. To adapt with the complex tracking environment, we introduce an aberrance repression regularization constraint to limit the drastic response changes, and propose a high-confidence updating strategy to optimize the overall tracking framework. The main contributions are summarized in the following three-folds:

• The AMRCF method is presented by simultaneously introducing adaptive spatial regularization, augmented memory regularization and aberrance repression regularization into the DCF framework. Combined with a dynamic appearance model update strategy, the overall tracking performance is improved in the case of partial/full occlusion, deformation and background clutter.
• Using the alternating direction method of multipliers (ADMM) [23] algorithm enables the model closure solution to be efficiently calculated.
• The overlap success and distance precision scores on four extensive benchmarks (i.e., OTB50, OTB100, TC128 and UAV123) verified that the proposed AMRCF has an excellent tracking performance comparable to state-of-the-art (SOTA) trackers.

2. Related Works

In this section, we mainly review the three categories of DCF tracking methods that are most relevant to our tracker, including the boundary effect-aware trackers, refined tracking models and high-confident updating schemes.

Due to the periodic assumption of the training samples generated by circular shifting not fully reflecting the sampling information, the tracking model is prone to over-fitting. To overcome the unwanted boundary effects, there are two main improvement directions for DCF. One is based on SRDCF, with the addition of spatial regularization, feature dimensionality reduction, feature interpolation and confidence map fusion [24,25,26]. Another is to adopt the BACF strategy of labelling around the target region, performing spatial feature restriction and adding constraint terms to the model [27,28,29,30]. Huang et al. [29] proposed a background suppressed correlation filter (BSCF) that incorporates all global background patches to enhance the tracking performance. ASRCF [31] incorporates SRDCF and BACF by employing the adaptive spatial regularization term, with the model adaptively penalizing the filter coefficients in the event of occlusion. A context-aware correlation filter CACF [32] takes into account global information, explicitly learning the background information around the target. Zha et al. [33] proposed the semantic-aware spatial regularization correlation filter by using spatial semantic maps to model regularization and feature selection.

To minimize the negative impacts of the model degradation and other issues, DCF has developed numerous improvements. The spatial–temporal regularized correlation filter (STRCF) [16] incorporates a temporal regularity term to minimize overfitting and reduce tracking failures due to target occlusion and distortion. Li et al. [34] proposed the augmented memory for correlation filter (AMCF) for jointly training the model with several frames of historical views and the current view to improve the stability of the model. LADCF [35] incorporates temporal consistency constraints into the model for enhanced tracker robustness and reduced model degradation. In recent work, many trackers incorporated spatial–temporal regularization [36,37,38], while the tracking performance has been significantly improved. Yu et al. [36] proposed a second-order spatial–temporal correlation filter (SSCF), which incorporates both the first-order and second-order data-fitting terms into the DCF framework. Hu et al. [37] merged the context-aware model into the STRCF tracker, thus expanding the target search domain and obtaining more discriminative information.

The output response maps are well reflective of the target state. In order to make better use of the information in the response maps, much work is performed in confidence map evaluation [1,39] and model updating strategies [20,21,22,40,41]. Bolme et al. [1] proposed the peak-to-sidelobe ratio (PSR) as the basis for the detection of confidence maps. Wang et al. [39] proposed the average peak-to-correlation energy (APCE) to predict the target state by comparing the change of APCE values for confidence maps. Fu et al. [20] proposed to control the update to the tracker by verifying the consensus score. Gan et al. [21] proposed a long-term correlation tracker, which can perform a long-term memory function by activating an online random fern classifier for re-detection when the PSR value of the confidence map is below a threshold. MUSTer [22] employs the Atkinson–Shiffrin memory model, which is divided into a long-term memory module for key points and a short-term memory module: when tracking fails or is obscured, the short-term module is updated. Ma et al. [40] proposed the bidirectional incongruity-aware correlation filter (BiCF), which predicts the tracking state by means of forward detection and backward relocation, and obtains a more robust model by suppressing bidirectional incongruity errors. Liu et al. [41] proposed a long-term tracker with multiple features and saliency redetection (MSLT), which consists of tracking-by-detection and redetection parts, and effectively improving the performance of the model.

3. Proposed Method

3.1. Review the BACF Tracker

Unlike traditional KCF [3] tracker, the BACF tracker expands the detection region and crops the real training samples using the binary mask matrix P. This allows BACF to obtain more high-quality negative samples, which greatly mitigates the boundary effect and achieves greater performance. The objective function of BACF is as follows:

$$\epsilon \left(h_c\right)=\frac{1}{2}\Vert y_c-{{\displaystyle \sum }}_{d=1}^D{x}_c^d\ast \left(P^T{h}_c^d\right){\Vert }_2^2+\frac{\lambda }{2}{{\displaystyle \sum }}_{d=1}^D\Vert h_c^d\Vert {}_2^2$$

(1)

where the subscript c indicates the current frame; $x\in {ℝ}^{M\times N\times D}$ denotes the D-dimensional feature maps of size $M\times N$ extracted from the input image; and $y\in {ℝ}^{M\times N}$ is represented as the ideal output of the filter. The feature maps x are correlated with the trained filter $h\in {ℝ}^{M\times N\times D}$ to obtain the final output ${\displaystyle \sum }_{d=1}^D{x}_c^d\ast h_c^d$. $\lambda$ is a regularization parameter.

Despite the good tracking performance of the BACF, there are still several issues that need to be further addressed: (1) the traditional DCF tracker, which utilizes a fixed learning rate to maintain and update the appearance model frame by frame, does not take into account the current object state, which made it difficult to adapt the complex tracking environment changes; (2) there is no reaction tactic to cope with anomalies and the fact that tracking targets can be easily lost; and (3) further optimization can be made by dealing with boundary effects.

3.2. Objective Function of AMRCF

Aiming at the weaknesses of the BACF tracker, we propose an augmented memory joint aberrance repressed correlation filter (AMRCF) to improve in terms of model stability and accuracy. The overall workflow and framework of the proposed AMRCF are represented in Figure 1 and Figure 2. The proposed AMRCF tracker based on the BACF tracker, introduces adaptive spatial regularization, augmented memory regularization and aberrance repression regularization to improve the overall tracking performance. Meanwhile, combining the update strategy of the appearance model, AMRCF can adapt to the changing tracking scenarios, and can then achieve more accurate and robust tracking results.

The objective function ε(h) of AMRCF is as follows:

$$\epsilon \left(h\right)=\frac{1}{2}\Vert y_c-{{\displaystyle \sum }}_{d=1}^D{x}_c^d\times \left(P^T{h}_c^d\right){\Vert }_2^2+{ℛ}_s+{ℛ}_m+{ℛ}_a$$

(2)

where ${ℛ}_s$, ${ℛ}_m$ and ${ℛ}_a$ denote the adaptive spatial regularization term, the augmented memory regularization term and the aberrance repressed regularization term, respectively.

3.2.1. Adaptive Spatial Regularization

The SRDCF model introduces the spatial regularization constraint, which is a negative Gaussian-shaped spatial weight vector, making it possible to have a strong response around the centre of the tracked object. Nevertheless, the fixed regularity fails to reflect the changing appearance of the target. Therefore, the proposed AMRCF introduced adaptive spatial regularization that penalizes the filter coefficients at the unreliable parts while approximating the spatial weights $w$ to the a priori reference weights $w_r$, preventing model degradation.

$${ℛ}_s=\frac{{\lambda }_1}{2}{{\displaystyle \sum }}_{d=1}^D\Vert w\odot h_c^d\Vert {}_2^2+\frac{{\lambda }_2}{2}\Vert w-w^r\Vert {}_2^2$$

(3)

where ${\lambda }_1$ and ${\lambda }_2$ denote the regularization coefficients of the respective regularization terms.

3.2.2. Augmented Memory Regularization

Due to the frame-by-frame update strategy of the appearance model, the historical view will be forgotten at an exponential rate as the number of iterations increases. The model will focus more on the most recent views, which reduce the anti-interference capability of the model. In this work, we introduced an augmented memory regularization constraint which utilized the perceptual hashing algorithm (PHA) [42] to select K-frame historical views with distinct differences and make them train the filter model together with the current view for enhancing memory and mitigating model degradation:

$${ℛ}_m=\frac{{\lambda }_3}{2}{\displaystyle \sum }_{k=1}^K\Vert y_k-{\displaystyle \sum }_{d=1}^D{x}_k^d\times \left(P^T{h}_c^d\right){\Vert }_2^2$$

(4)

where $x_k^d$ and $y_k$ denote the d-channel training sample of the k-th view in the memory sequence and its ideal output, respectively. $h_c$ denotes the filter model of the current frame. ${\lambda }_3$ denotes the regularization coefficient. K indicates the capacity of the memory queue for storing the history view.

PHA generated a unique “fingerprint” for each view by comparing the fingerprint differences between the before and after frame views to decide whether to update and maintain the memory queue.

The initial view is converted to grayscale and transformed to the frequency domain using the discrete cosine transform (DCT). Only the low frequency region $A\in {ℝ}^{W\times W}$. with high energy density is retained. Then, each point $h_{i,j}$ is compared with the average of all elements to obtain the hash matrix $P$:

$$p_{i,j}=\{\begin{array}{cc}1,& \hfill h_{i,j}>\frac{1}{W^2}{\displaystyle \sum _{i=1}^W}{\displaystyle \sum _{j=1}^W}{h}_{i,j}\\ 0,& \hfill others\end{array}$$

(5)

By XOR operation, the difference score $S$ between the current frame $P^c$ and the latest view $P^k$ in the memory queue is derived:

$$S=\frac{1}{W^2}{\displaystyle \sum }_{i=1}^W{\displaystyle \sum }_{j=1}^W\left(p_{ij}^c\oplus p_{ij}^k\right)$$

(6)

When the obtained score is greater than a certain threshold $\tau$, indicating that there is a significant difference between the current frame object and the latest frame in the memory queue, then the memory queue will be updated according to the “first-in-first-out” principle.

3.2.3. Aberrance Repression Regularization

Ideally (i.e., without much change in the object appearance), the response of two adjacent frames does not tend to change much. However, abrupt changes in appearance caused by background clutter, target occlusion, etc., will cause response anomalies. As aberration occur, the similarity between the output response maps $M_1$ and $M_2$ suddenly drops, while the Euclidean distance between $M_1$ and $M_2$ will become larger. In this work, we adopted aberrance-suppressed regularization to restrict the response variation, which can effectively limit the response variation:

$${ℛ}_a=\frac{{\lambda }_4}{2}\Vert {\displaystyle \sum }_{d=1}^D{x}_{c-1}^d\times \left(P^T{h}_{c-1}^d\right)\left[{\psi }_{p,q}\right]-{\displaystyle \sum }_{d=1}^D{x}_c^d\times \left(P^T{h}_c^d\right){\Vert }_2^2$$

(7)

where $p$ and $q$ denote the difference in the peak position between the response maps and the shift operation $\left[{\psi }_{p,q}\right]$ to make the peak points of two response maps overlap each other. ${\lambda }_4$ denotes the regularization coefficient.

3.3. Optimization

Introducing auxiliary variables $\widehat{G}=[{\widehat{g}}^1,{\widehat{g}}^2,\dots ,{\widehat{g}}^D]({\widehat{g}}^d=\sqrt{T}F{P}^T{h}^d, d=1,2,\dots ,D)$, (where the superscript $ˆ$ denotes the discrete Fourier operator and $F$ denotes the standard orthogonal matrix), which translated Equation (2) into the frequency domain representation according to Parseval’s theorem:

$$\begin{array}{c}E(H,\widehat{G},w)=\frac{1}{2T}\Vert {\widehat{y}}_c-{\displaystyle {\displaystyle \sum }_{d=1}^D}{\widehat{x}}_c^d\odot {\widehat{g}}_c^d{\Vert }_2^2+\frac{{\lambda }_1}{2}{\displaystyle {\displaystyle \sum }_{d=1}^D}\Vert w\odot h_c^d\Vert {}_2^2+\frac{{\lambda }_2}{2}\Vert w-w^r\Vert {}_2^2\\ +\frac{{\lambda }_3}{2T}{\displaystyle {\displaystyle {\displaystyle \sum }_{k=1}^K}}\Vert {\widehat{y}}_k-{\displaystyle {\displaystyle \sum }_{d=1}^D}{\widehat{x}}_k^d\odot {\widehat{g}}_c^d{\Vert }_2^2+\frac{{\lambda }_4}{2}\Vert M_{c-1}\left[{\psi }_{p,q}\right]-{\displaystyle {\displaystyle \sum }_{d=1}^D}{\widehat{x}}_c^d\odot {\widehat{g}}_c^d{\Vert }_2^2\end{array}$$

(8)

where the output response of the (c − 1)-th frame is rewritten as $M_{c-1}={{\displaystyle \sum }}_{d=1}^D{x}_{c-1}^d\times \left(P^T{h}_{c-1}^d\right)$.

Rewrite Equation (8) using the augmented Lagrange multiplier method (ALM) as:

$$ℒ(H,\widehat{G},w,\widehat{\zeta })=E(H,\widehat{G},w)+{\widehat{\zeta }}^T({\widehat{g}}_c^d-\sqrt{T}F{P}^T{h}_c^d)+\frac{\mu }{2} \Vert {\widehat{g}}_c^d-\sqrt{T}F{P}^T{h}_c^d{\Vert }_2^2$$

(9)

where the penalty factor $\mu$ and the matrix of auxiliary variables ${\widehat{\zeta }}^T={[{\widehat{\zeta }}^{1T},{\widehat{\zeta }}^{2T},\dots , {\widehat{\zeta }}^{DT}]}^T$ are introduced.

In order to obtain closed solutions, Equation (9) is solved in steps via ADMM, for which we decompose it into three sub-problems to solve, and all three sub-problems have closed solutions.

Subproblem ${\mathit{h}}_{\mathit{c}+1}^{\ast }$: If $\widehat{G}$, $w$ and $\widehat{\zeta }$ are given, then $h$ yields:

$$\begin{array}{cc}\hfill h_{c+1}^{\ast }& ={\left(T\mu +{\lambda }_1{w}^{T}w\right)}^{-1}\left(\sqrt{T}P{F}^T\widehat{\zeta }+\mu \sqrt{T}P{F}^T{\widehat{g}}_c\right)\hfill \\ \hfill & ={\left(\mu I_c+\frac{{\lambda }_1{w}^{T}w}{T}\right)}^{-1}\left(\zeta +\mu g_c\right)\hfill \end{array}$$

(10)

where $\zeta$ and $g$ can be easily obtained via the Fourier inverse operation on $\widehat{\zeta }$ and ${\widehat{g}}_c$.

$$\{\begin{array}{c}\zeta =\frac{1}{\sqrt{T}}P{F}^T\widehat{\zeta }\\ g_c=\frac{1}{\sqrt{T}}P{F}^T{\widehat{g}}_c\end{array}$$

(11)

Subproblem ${\widehat{\mathit{G}}}^{\ast }$: If given $w$, $\widehat{\zeta }$ and $h_{c+1}^{\ast }$, since each channel information is relatively independent for the sample ${\widehat{x}}_c^d$, the subproblem ${\widehat{G}}^{\ast }$ can be further split into $d=\left\{1,2,\dots ,D\right\}$ smaller problems, each of which can be denoted as:

$${\widehat{G}}^{\ast }\left(d\right)=\underset{{\widehat{G}}^{\ast }\left(d\right)}{argmin}\left\{\begin{array}{c}\frac{1}{2T}\Vert {\widehat{y}}_c-{\widehat{x}}_c^T{\widehat{g}}_c{\Vert }_2^2+\frac{{\lambda }_3}{2T}{\displaystyle \sum _{k=1}^K}\Vert {\widehat{y}}_k-{\widehat{x}}_k^T{\widehat{g}}_c{\Vert }_2^2\\ +\frac{{\lambda }_4}{2}\Vert M_{c-1}\left[{\psi }_{p,q}\right]-{\widehat{x}}_c^T{\widehat{g}}_c{\Vert }_2^2\\ +{\widehat{\zeta }}^T\left( {\widehat{g}}_c-\sqrt{T}F{P}^T{h}_c\right)+\frac{\mu }{2} \Vert {\widehat{g}}_c^d-\sqrt{T}F{P}^T{h}_c{\Vert }_2^2\end{array}\right\}$$

(12)

The closed solution of ${\widehat{G}}^{\ast }\left(d\right)$ can be solved as:

$$\begin{array}{c}{\widehat{G}}^{\ast }\left(d\right)={\left[\left({\lambda }_4+\frac{1}{T}\right){\widehat{x}}_c{\widehat{x}}_c{}^T+\frac{{\lambda }_3}{T}{\displaystyle \sum }_{k=1}^K{\widehat{x}}_k{\widehat{x}}_k{}^T+\mu \right]}^{-1}\\ \left(\frac{1}{T}{\widehat{x}}_c{\widehat{y}}_c+\frac{{\lambda }_3}{T}{\displaystyle \sum }_{k=1}^K{\widehat{x}}_k{\widehat{y}}_k+{\lambda }_4{\widehat{x}}_c{M}_{c-1}-\widehat{\zeta }+\mu {\widehat{h}}_c\right)\end{array}$$

(13)

Further reducing the computational effort by using the Sherman–Morrison formula, i.e., ${\left(u{v}^T+A\right)}^{-1}=A^{-1}-\frac{A^{-1}u{v}^T{A}^{-1}}{1+v^T{A}^{-1}u}$ (here $u$ and $v$ are two column vectors and $u{v}^T$ is a rank one matrix), here set $u=v^T={\widehat{x}}_c, A=\frac{{\lambda }_3}{T}{\displaystyle \sum }_{k=1}^K{\widehat{x}}_k{\widehat{x}}_k{}^T+\mu$, $B={\lambda }_4+\frac{1}{T}$. ${\widehat{G}}^{\ast }\left(d\right)$ is further derived as:

$$\begin{array}{c}{\widehat{G}}^{\ast }\left(d\right)=\frac{1}{A}\left(\frac{1}{T}{\widehat{x}}_c{\widehat{y}}_c+\frac{{\lambda }_3}{T}{\displaystyle \sum }_{k=1}^K{\widehat{x}}_k{\widehat{y}}_k+{\lambda }_4{\widehat{x}}_c{M}_{c-1}-\widehat{\zeta }+\mu {\widehat{h}}_c\right)\\ -\frac{{\widehat{x}}_c}{Ab}\left[\frac{{\widehat{S}}_{xx}{\widehat{y}}_c}{T}+\frac{{\lambda }_4{\widehat{S}}_{xc}}{T}+{\lambda }_3{\widehat{S}}_{xx}{M}_{c-1}-{\widehat{S}}_{x\zeta }+\mu {\widehat{S}}_{xh}\right]\end{array}$$

(14)

where $b=\frac{A}{B}+{\widehat{x}}_c{}^T{\widehat{x}}_c$, ${\widehat{S}}_{xx}={\widehat{x}}_c{}^T{\widehat{x}}_c$, ${\widehat{S}}_{xc}={\widehat{x}}_c{}^T{\displaystyle \sum }_{k=1}^K{\widehat{x}}_k{\widehat{y}}_k$, ${\widehat{S}}_{x\zeta }={\widehat{x}}_c{}^T\widehat{\zeta }$, ${\widehat{S}}_{xh}={\widehat{x}}_c{}^T{\widehat{h}}_c$.

Subproblem ${\mathit{w}}^{\ast }$: If $h$ is given, then $w$ can be solved for:

$$\begin{array}{c}{w}^{\ast }={\left({\lambda }_1{\displaystyle {\displaystyle \sum }_{d=1}^D}{h}_c^d\odot h_c^d+{\lambda }_{2}I\right)}^{-1}{\lambda }_2{w}^r\\ =\frac{{\lambda }_2{w}^r}{{\lambda }_1{{\displaystyle \sum }}_{d=1}^D{h}_c^d\odot h_c^d+{\lambda }_{2}I}\end{array}$$

(15)

Update of Lagrange multipliers $\widehat{\zeta }$: Updates to $\widehat{\zeta }$ are made by:

$${\widehat{\zeta }}_{i+1}={\widehat{\zeta }}_i+\mu \ast \left({\widehat{G}}_{i+1}-{\widehat{h}}_{i+1}\right)$$

(16)

where the subscripts $i$ and $i+1$ denote the $i$-th and $\left(i+1\right)$-th iterations, respectively. ${\widehat{G}}_{i+1}$ and ${\widehat{h}}_{i+1}$ denote the subproblem solutions obtained in the i + 1-th iteration, respectively. Here, the canonical constant ${\mu }^{i+1}=\min \left({\mu }_{max},\beta {\mu }^i\right)$.

Complexity Analysis: Since each pixel is relatively independent, we need to solve the D*MN subproblems, where D represents the number of channels. Since the subproblem G requires the FFT and inverse FFT transformation in each iteration, so the computational complexity is O(DMNlog(MN)). Meanwhile, the computational complexity of both subproblems h and w is O(DMN). Therefore, the overall complexity of model is O(LDMNlog(MN)) when the number of iterations is L. It is worth remarking that ${\displaystyle \sum }_{k=1}^K{\widehat{x}}_k{\widehat{y}}_k$ and ${\displaystyle \sum }_{k=1}^K{\widehat{x}}_k{\widehat{y}}_k$ in Equation (14) represent the actual view information stored in the memory queue, and are constant terms that do not require additional computational resources.

3.4. Detection

To strengthen the effects of the reliable channels on the final response results, different channels are assigned corresponding channel weights. The PSR value, as the evaluation criterion for the confident map [1], is adopted as the reference for the performance representation of the different channels, and the channel weights are determined and updated as follows:

$$C_{c+1}^d=\left(1-\gamma \right)C_c^d+\gamma \frac{PSR\left({\widehat{h}}_c^{d\ast }\odot {\widehat{x}}_c^d\right)}{{{\displaystyle \sum }}_{d=1}^{D}PSR\left({\widehat{h}}_c^{d\ast }\odot {\widehat{x}}_c^d\right)}$$

(17)

where $\gamma$ denotes the learning rate of the channel weights. $PSR\left(·\right)=\raisebox{1ex}{$\left(\max \left(R\right)-\mu \right)$}\!\left/ \!\raisebox{-1ex}{$\sigma $}\right.$, $R$ denotes the output response map, $\mu$ and $\sigma$ denote the mean and mean squared deviation of the response map, respectively.

The final output response $R_c$ can be derived as follows:

$$R_c={ℱ}^{-1}\left({\displaystyle \sum }_{d=1}^D{C}_c^d{\widehat{h}}_{c-1}^{d\ast }\odot {\widehat{x}}_c^d\right)$$

(18)

where ${ℱ}^{-1}$ denotes the Fourier inverse operation. The individual channel output responses are multiplied by the corresponding channel weights to the final output response map, and the maximum point is the target position.

3.5. Model Update

To adapt to the changing tracking environment, we use the maximum peak and APCE values as a basis for determining the reliability of the output response. As shown in Figure 3, ideally, the output response approximates a two-dimensional normal distribution with a relatively flat response and prominent main peak, and the corresponding APCE values are in the higher range. When anomalies occur such as occlusion, blurring, and illumination variation, the output response map fluctuates dramatically and the maximum response value decreases, while the APCE value will also decrease. Furthermore, the maximum peak position may not be the target position. Therefore, the tracking results are considered reliable when both the maximum peak and APCE values are above a certain ratio of the respective historical averages, as shown in Figure 4.

$$\{\begin{array}{c}{F}_{t,max}\ge {\beta }_1\frac{1}{t-1}{\displaystyle {\displaystyle \sum }_{i=1}^{t-1}}{F}_{i,max}\\ APC{E}_t\ge {\beta }_2\frac{1}{t-1}{\displaystyle {\displaystyle \sum }_{i=1}^{t-1}}APC{E}_i\end{array}$$

(19)

where $F_{t,max}$ and $APC{E}_t$ denote the maximum peak value and average APCE value of the output response map at frame t, respectively. $APCE=\frac{{\left|F_{max}-F_{min}\right|}^2}{\mathrm{mean}\left({{\displaystyle \sum }}_{w,h}{\left(F_{w,h}-F_{min}\right)}^2\right)}$, $F_{max}$ and $F_{min}$ indicate the maximum and minimum value of the response map, respectively.

Depending on the object state, we selected different learning rates to maintain the appearance model. Considering that the target may be in an unreliable state for a long time, in this case, the learning rate should be further reduced. The learning rate is determined as follows:

$$l{r}_{new}=\{\begin{array}{cc}\alpha -\beta \ast l{r}_{old}& \mathrm{Reliable} \mathrm{state}\\ 0.012& \mathrm{Unreliable} \mathrm{state}\\ 0.003& \ast \end{array}$$

(20)

where $\alpha$ and $\beta$ are set to 0.0175 and 0.1, respectively, as empirical constants. $\ast$ indicates the condition of “three consecutive frames judged to be unreliable state”. As the target state changes from unreliable to reliable, the learning rate forms a small pulse and quickly stabilizes to $\frac{\alpha }{1+\beta }$.

The appearance model ${\widehat{x}}_c{}^{model}$ updates as follows:

$${\widehat{x}}_{c+1}^{model}=\left(1-l{r}_{new}\right){\widehat{x}}_c{}^{model}+l{r}_{new}{\widehat{x}}_c$$

(21)

where ${\widehat{x}}_c^{model}$ and ${\widehat{x}}_{c+1}^{model}$ represent the appearance model of the current frame and the next frame, respectively.

The overall workflow of the proposed AMRCF algorithm is as follows (Algorithm 1):

Algorithm 1: Augmented memory joint aberrance repressed correlation filter (AMRCF)

Input: First frame state of the sequence (i.e., target position and scale information);

Output: Target position at frame t;

Initialize tracker model hyperparameters.

for t = 1: end do

Training

Extract multi-channel feature maps $x_t^d$

Calculate the hash matrix $p_t$

if t = 1 then

Initialize the FIFO memory sequence.

Initialize the channel weight model $C_t^d=1/D$

Initialize the appearance model.

else

Calculate the score between $p_t$ and $p_{t-1}$.

if score > $\tau$ then

Update the FIFO memory queue.

end if

Store the hash matrix $p_t$.

end if

Optimize the filter model $h_t$ via Equations (10) and (14)–(16) for L iterations

Detecting

Crop multi-scale search regions centered at $P_t$ with S scales based on the bounding box at frame t + 1.

Extract multi-channel feature maps $x_{t+1}^d$

Use Equation (18) to final response output map $R_r, \left(r=1, 2, \dots ,S\right)$.

Estimate the target position $T_{t+1}$ and scale $s_{t+1}$ from the maximum value of the response maps.

Updating

Use Equation (17) to update the channel weight model $C_{t+1}$

Use Equation (20) to Calculate the learning rate $l_{new}$.

Use Equation (21) to update the appearance model.

end for

4. Experiments

In this section, we adopt the one-pass evaluation (OPE) criterion on four widely used benchmark datasets (OTB50 [43], OTB100 [44], Temple-Color 128 [45] and UAV123 [46], respectively, with 269 challenge image sequences over 220k images) to evaluate the proposed algorithm and several SOTA trackers, including BACF [15], SRDCF [13], AutoTrack [47], STRCF [16], ARCF [19], ARCF_H [19], MUSTER [22], BiCF [40], fDSST [6], Staple [9], LADCF_HC [35], AMCF [34], and ECO_HC [25]. The evaluation criteria include the overlap success rate (SR) and distance precision (DP), while the tracking speed is measured in terms of frames per second (FPS).

4.1. Implementation Details

The experiments involved were conducted in the same configuration (CPU Intel i7-7700 3.60 GHz 8.00 GB RAM and GPU NVIDIA GT730). For the reference tracker, the model structure and parameters were obtained from publicly available sources without any modifications. For the proposed AMRCF, the parameters are set as follows: The adaptive spatial regularization coefficients are ${\lambda }_1=0.2$, ${\lambda }_2=0.001$, the augmented memory regularization coefficient ${\lambda }_3=0.05$, the aberrance repression regularization coefficient ${\lambda }_4=0.233$, the learning rate of the channel weight model $\gamma =0.018$, the initial appearance model learning rate is $l{r}_{init}=0.013$, the length of the memory queue $K=5$, the selection threshold $\tau$ for the memory sequence is set to 0.5, and the reliability judgement conditions of ${\beta }_1$ and ${\beta }_2$ are 0.3 and 0.6, respectively. We predict target placement using HOG, CN and Grayscale features, with scale evaluation using only five scales of HOG features.

4.2. Evaluation of the OTB Benchmark

4.2.1. Overall Performance

The OTB benchmark dataset contains 50 and 100 annotated video sequences with 11 different challenge attributes, i.e., motion blur (MB), illumination variation (IV), out-plane rotation (OPR), in-plane rotation (IPR), scale variation SV, deformation (DEF), fast motion (FM), out-of-view (OV), occlusion (OCC), background clutter (BC) and low resolution (LR). The overall results are presented in Figure 5 and Figure 6. It can be seen that the performance of the proposed tracker is fully comparable to most good trackers. The proposed AMRCF gained 88.9% and 84.5% scores for the success rate and DP on the OTB50 benchmark while gained 86.2% and 81.8% scores on the OTB100 benchmark. The proposed AMRCF increased the success rate and DP by 5% and 3.7%, respectively, for OTB50; and by 4.6% and 5% for OTB100 compared to the baseline tracker BACF; while the proposed AMRCF increased the overlap success rate and DP results on the OTB100 benchmark dataset by 8.6% and 9.9%, respectively, against the spatial regularization-based SRDCF tracker; by 9.6% and 9.7%, respectively, against the augmented memory-based AMCF tracker; and by 5.7% and 7.1%, respectively, against the aberrance repression-based ARCF tracker.

The performance of the mainstream trackers on the OTB100 benchmark in terms of accuracy and tracking speed in

Table 1. Accuracy and tracking speed performance of the top-8 trackers on the OTB100 benchmark.

Trackers	STRCF	ECO_HC	LADCF_HC	BACF	ARCF	AMCF	SRDCF	Ours
DP (%)	86.4	84.5	86.1	81.6	80.7	76.6	77.6	86.2
SR (%)	80.0	77.2	80.6	76.8	74.7	72.1	71.9	81.8
FPS	25.57	51.06	20.23	39.49	17.07	34.44	7.22	12.56

Note: The top and second ranked outcomes are shown with red and green.

. The proposed AMRCF achieves excellent performance in terms of success rate and accuracy. Although the proposed algorithm greatly improves the accuracy of the basic tracker, it sacrifices the tracking speed due to the large number of additional operations added to the baseline BACF.

4.2.2. Attribute Evaluation

To further demonstrate the tracking performance of the proposed AMRCF in the complex real-world environment, Figure 7 shows the attribute-based experimental results on the OTB100 benchmark. It is clear that most of the attribute-based results of our proposed algorithm obtained the best performance compared to the SOTA trackers, especially in the background clutter, motion blur, fast motion, where the success rate and distance precision reached 85.7% and 85.9%, 80.9% and 82%, 77.5% and 82%, all ranking the best results, while in the out-of-view, fast motion, illumination variation, in-plane-rotation also obtained excellent performance.

4.2.3. Qualitative Evaluation

To further demonstrate the superiority of the proposed AMRCF, Figure 8 shows the results of a qualitative comparison between the AMRCF and seven SOTA trackers, including STRCF, LADCF_HC, ECO_HC, ARCF, SRDCF, BACF and AMCF. The selected test sequences are Soccer, Dragonbaby, Box, Bolt 2, Girl 2 and Basketball (from top to bottom), where each sequence contains at least multiple challenging attributes, including OCC (Soccer, Dragonbaby, Box, Girl 2 and Basketball), BC (Soccer, Box, Bolt 2 and Basketball), FM (Soccer and Dragonbaby), MB (Soccer, Dragonbaby, Box and Girl 2), DEF (Bolt 2, Girl 2 and Basketball), etc. The results show that the proposed AMRCF was able to perform the tracking task relatively well in a variety of scenarios. Significantly, the proposed tracker demonstrates its superior performance in the “Soccer” sequence with a cluttered background or in the “Girl2” and “Box” sequences which are partially or completely obscured. In the “Soccer” sequence, despite massive background interference, deformation and occlusion problems in frames 100–200, AMRCF was still able to accomplish the subsequent tracking task while most trackers lost the target. In frames 105–120 of the “Girl 2” sequence, the target is lost due to full occlusion. However, when the target reappears in the 120-frame view, AMRCF is still able to redetect and achieve stable tracking. Comparing the performance of the baseline BACF tracker, this further validates that the improvements in model structure and high-confidence updating strategy of the proposed tracker are not redundant.

4.3. Evaluation of the TC128 Benchmark

In order to approximate real tracking scenes, unlike the OTB dataset which includes 25% grey scale sequences, the Temple Color benchmark collects 128 colour videos containing 27 object categories. An overview of the experimental results for the TC128 benchmark is presented in Figure 9. The proposed AMRCF achieves excellent tracking performance with the success rate and DP of 69.7% and 75.8%, respectively. Comparing the experimental results of the baseline tracker BACF (whose success rate and DP are 61.3% and 65.3%, respectively), the proposed AMRCF outperforms the performance by 8.4% and 11.2%, respectively. In comparing with the SRDCF, ARCF and AMCF tracker, the performance of the proposed method gains 9.5%, 5.6% and 9.2% in DP score.

4.4. Evaluation of the UAV123 Benchmark

The UAV123 benchmark consists of 91 image sequences captured by low-altitude UAVs. In contrast with the OTB100, the UAV123 has the challenges of a wide variation of shooting angles, very long video sequences, small targets and targets that disappear completely from the view for an extended time, which poses an even greater challenge for the tracker. On the one hand, the tracker model needs to be robust and able to follow the target again after complete occlusion, and on the other hand, the model needs to be quickly updated to accommodate rapid changes in shooting angles. An overview of the experimental results for the UAV123 benchmark is presented in Figure 10. As it can be seen, the proposed AMRCF is ranked first in the success rate (58.4%) and third in DP (69%), respectively. Compared to the baseline tracker, the proposed tracker is outperformed by 2.9% and 3% in terms of success rate and DP.

Figure 11 shows the success plots for nine representative attributes on UAV123 benchmark, including Camera Motion (CM), Scale Variation (SV), Illumination Variation (IV), Aspect Ratio Change (ARC), Fast Motion (FM), Viewpoint Change (VC), Similar Object (SO), Partial Occlusion (PO) and Out-of-View (OV). Compared with other SOTA trackers, the proposed AMRCF performs remarkably well in these challenging attributes. Four of the nine attribute results ranked first, including CM (58.1%), SV (54%), ARC (48.2%), FM (43.9%), and the rest maintained the top three positions, slightly worse than the first, which proved that AMRCF maintained a high tracking performance for dealing with various tracking scenarios. This can also be attributed to the synergy of multiple regularization constraints and the dynamic model update strategy, which greatly mitigates the boundary effects, response distortions and filter degradation while providing a robust appearance model.

4.5. Compared with Deep Feature-Based Trackers

In this section, we discuss the effectiveness of the proposed AMRCF by comparing the deep feature-based tracking algorithms, including LADCF [35], DeepSTRCF [16], SRDCFdecon [48], SiamFC3s [49], SiamFC [49], DCFNet [50] and CFNet [51]. From

Table 2. Accuracy and tracking speed performance of the deep learning trackers on the OTB50 benchmark.

Trackers	LADCF	DeepSTRCF	SRDCFdecon	SiamFC3s	SiamFC	DCFNet	CFNet	Ours
DP (%)	87.2	86.6	81.4	77.9	72.0	84.2	72.4	85.9
SR (%)	88.4	87.3	87.0	80.9	77.2	87.7	78.1	88.9
FPS	5.93	3.57	2.41	9.82	12.82	99.37	5.7	13.13

Note: The first, second and third ranked outcomes are shown in red, green and blue.

, we can see that our algorithm ranks third in terms of success rate and first in terms of precision on the OTB50 benchmark. In terms of tracking speed, however, we have a relatively obvious advantage over deep feature-based algorithms. The attribute-based experimental results, presented in

Table 3. Precision scores of AMRCF and deep feature-based trackers on the OTB50 benchmark.

	IV	OPR	SV	OCC	DEF	MB	FM	IPR	OV	BC	LR
LADCF	79.0	87.6	84.9	89.8	87.3	74.8	79.2	81.5	79.4	81.4	55.4
DeepSTRCF	78.0	86.3	82.7	88.4	86.9	74.6	75.9	79.8	81.1	79.9	56.9
SRDCFdecon	81.8	85.8	83.3	86.1	84.2	79.0	78.3	80.6	78.5	84.6	52.7
SiamFC3s	70.9	78.8	79.6	80.2	74.3	69.8	72.3	74.3	78.0	73.2	65.9
SiamFC	67.6	74.7	77.3	68.0	70.1	52.5	66.9	71.7	64.3	72.3	48.9
DCFNet	82.7	86.7	88.3	87.0	83.4	74.3	77.4	83.9	80.3	82.8	64.2
CFNet	70.3	79.3	78.6	77.4	73.2	56.7	56.8	73.5	57.1	70.9	57.9
Ours	84.5	88.5	84.4	88.0	88.6	81.4	79.3	85.6	81.6	86.6	56.3

Note: The best results are shown in bold.

, show that the proposed AMRCF achieves the excellent performance in precision with other deep feature-based trackers under IV, OPR, DEF, MB, FM, IPR, OV and BC challenge attributes, which further demonstrate the advantages of our tracker architecture.

4.6. Ablation Studies and Effectiveness Discussion

In this section, we discuss the effectiveness of the different modules in the proposed the AMRCF and compare them with the baseline BACF tracker. Moreover, AMRCF-AS adds the adaptive spatial regularization module (AS) to the baseline BACF tracker. However, AMRCF-AM, AMRCF-AR and AMRCF-CL add the augmented memory module (AM), aberrance repression module (AR) and channel weight with learning rate update module (CL) to the BACF, respectively. AMRCF-AS-AM and AMRCF-AS-AM-CL are obtained by adding AM module and CL module to AMRCF-AS in turn. The AMRCF method is our complete tracking framework, which incorporates all modules.

The overall ablation experimental results are presented in

Table 4. Overall evaluation performance on the OTB100 and UAV123 benchmarks with the progressive addition of different modules on the baseline BACF tracker.

Trackers	OTB100 Benchmark		UAV123 Benchmark		Average
	Success (%)	Precision (%)	Success (%)	Precision (%)	Success (%)	Precision (%)
BACF (Baseline)	76.8	81.6	55.5	66.0	66.2	73.8
AMRCF-AS	79.3	83.4	57.3	68.0	68.3	75.7
AMRCF-AM	79.4	84.0	58.1	68.3	68.8	76.2
AMRCF-CL	81.4	85.7	58.0	70.1	69.7	77.9
AMRCF-AR	81.5	85.4	57.3	68.3	69.4	76.9
AMRCF-AS-AM	80.7	85.5	58.3	68.6	69.5	77.1
AMRCF-AS-AM-CL	81.6	85.9	58.2	69.3	69.9	77.6
AMRCF	81.8	86.2	58.4	69.0	70.1	77.6

Note: The best results are shown in bold.

. It can be seen that the original baseline BACF tracker achieves scores of 66.2% and 73.8% in the average success and precision. Benefiting from the AS module, AMRCF-AS outperforms the baseline in terms of the average accuracy and precision by 2.1% and 1.9%, respectively. Meanwhile, AMRCF-AM also outperforms the baseline in terms of the average accuracy and precision by 2.6% and 2.4%, respectively, which indicates that the selective recollection of history views is effective for the overall tracker performance. For the average accuracy and precision, AMRCF-AR increased by 3.2% and 3.1% while AMRCF-CL increased by 3.5% and 4.1%. The above experimental results all indicate the effectiveness of each module for the tracking framework. Furthermore, the performance of the baseline BACF improves with the introduction of key modules. Ultimately, the average success and precision scores of the proposed AMRCF, which incorporates all modules, exceeded the baseline tracker by 3.9% and 3.8%, respectively.

5. Conclusions

In this paper, based on the background-aware correlation filter, we proposed a novel augmented memory joint aberrance repressed correlation filter (AMRCF) for visual tracking. By introducing different regularizers (adaptive spatial regularization, augmented memory regularization and aberrance suppression regularization) and combining with a high-confidence updating strategy, the adverse effects caused by boundary effects, model degradation and response anomalies are effectively mitigated, making the trained model adaptable to changing tracking scenarios. The ADMM algorithm is employed to reduce the computational complexity during model optimization. In addition, extensive comparative experiments on four well-known benchmarks indicate that the proposed AMRCF tracker achieved a tracking performance comparable to 14 SOTA trackers, especially in environments such as background clutter, motion blur, out-of-view and fast motion. However, as the proposed tracker cannot achieve real time in terms of tracking speed, subsequent work will be carried out on increasing the speed of the tracker.