Adaptive Kalman Filter for Real-Time Visual Object Tracking Based on Autocovariance Least Square Estimation

Abstract

Real-time visual object tracking (VOT) may suffer from performance degradation and even divergence owing to inaccurate noise statistics typically engendered by non-stationary video sequences or alterations in the tracked object. This paper presents a novel adaptive Kalman filter (AKF) algorithm, termed AKF-ALS, based on the autocovariance least square estimation (ALS) methodology to improve the accuracy and robustness of VOT. The AKF-ALS algorithm involves object detection via an adaptive thresholding-based background subtraction technique and object tracking through real-time state estimation via the Kalman filter (KF) and noise covariance estimation using the ALS method. The proposed algorithm offers a robust and efficient solution to adapting the system model mismatches or invalid offline calibration, significantly improving the state estimation accuracy in VOT. The computation complexity of the AKF-ALS algorithm is derived and a numerical analysis is conducted to show its real-time efficiency. Experimental validations on tracking the centroid of a moving ball subjected to projectile motion, free-fall bouncing motion, and back-and-forth linear motion, reveal that the AKF-ALS algorithm outperforms a standard KF with fixed noise statistics.

1. Introduction

Visual object tracking (VOT) is a fundamental task in computer vision and has extensive applications including autonomous vehicles [1], video surveillance [2], robot vision [3], and human-computer interaction [4]. Specifically, in autonomous vehicle systems and robotics, robust and efficient VOT algorithms that identify and track nearby vehicles and pedestrians are essential for real-time navigation, obstacle avoidance, and environment perception, ensuring safe and efficient operation in dynamic scenarios. The essence of VOT is to accurately track the states (e.g., the centroid, the bounding box, or the contour) of an object of interest through successive frames within a video stream. This involves two key steps: object detection which identifies the presence and location of the object, and object tracking which associates the object and generates the trajectory of the object throughout the frame sequence.

The challenges faced in VOT are accentuated in the context of autonomous systems. These systems must perform reliably amidst non-stationary conditions caused by variable lighting, weather conditions, diverse terrains, and unstructured environments. Furthermore, in robotics, VOT is tested against robot motion, varying perspectives, and interactions with objects that can introduce additional complexities. The object may change poses, deform, become occluded, or disappear temporarily from the field of view [5], resulting in high model and measurement noise in the object tracking step. Regarding the uncertainty (e.g., sudden or drastic changes in object appearance) as the noise, the approach to the two above challenges depends heavily on the real-time accurate estimation of the noise statistics, especially the accurate noise covariance estimation. Therefore, the real-time VOT under uncertain noise covariance problem is addressed in this paper, which can account for the varying model and measurement noise covariance’s influence on object detection and tracking.

1.1. Object Detection

The background subtraction (BS) technique is a widely used object detection technique for extracting moving foreground objects from video against a static background. In an autonomous vehicle or robot’s vision systems, the choice of BS technique can significantly influence performance due to several factors, such as illumination variations, occlusion, shape deformation, scale variation, and background clutter. According to the model and computational complexity, the BS techniques can be classified into three broad categories: simple BS, statistical BS, and neural networks (NN) based BS [6,7,8,9].

• The simple BS techniques including the frame differencing [10], average filtering [11,12], median filtering [13,14], and histogram over time [15], model the background simply, often by just using the previous frames or an average/median of the recent frames. These are fast and easy to implement but lack adaptability to changes in the background.
• The statistical BS techniques including the mixture of Gaussians (MoG) [16,17], kernel density estimation [18,19,20], support vector models [21,22], and principal component analysis [23,24], build a statistical model of the background and classify pixels based on the model. They model the background more robustly using the history of pixels but are prone to the setting parameters and models.
• NN-based BS techniques including the radial basis function NN [25], self-organizing NN [26,27], the convolutional NN [28,29], and the generative adversarial networks (GAN) [30,31], learn the specialized NN architectures that can adapt to changes in the background model to detect foreground over time. These can learn complex representations of background appearance and maintain robust models of multi-modal backgrounds, but they require large training datasets and have expensive computation loads to train and run, which may not be available in a real-time environment.

Dynamic backgrounds with movement from objects can be erroneously marked as foreground if the model cannot adapt swiftly enough, leading to noisy segmentation and false detections. Therefore, the simple frame difference-based BS technique with the adaptive threshold (AT) method is appealing in the real-time VOT application due to its simplicity, efficiency, and robustness [32]. In this method, the threshold is adjusted based on the pixel’s intensity variations over some time, e.g., the threshold should be smaller for regions with low contrast.

1.2. Object Tracking

Recent developments of VOT have been witnessed to solve the VOT problem from the representation (e.g., correlation filter) based tracker [33,34], discriminate and generative tracker [35], to deep Siamese networks based tracker [36,37]. Conventional representation-based VOT methods rely on target appearance models based on handcrafted visual features and are prone to inaccuracies from background interference and occlusion. The discriminative and deep Siamese networks-based VOT methods are more robust but may fail to update adequately to sudden variations of the object in real-time applications due to high computation complexity.

To achieve robust and real-time VOT, various recursive Bayesian filtering methods like the mean-shift (MS) tracker [38,39], the Kalman filter (KF) tracker [40,41,42], and particle filter (PF) tracker [43,44,45] are commonly developed. Among the above, the KF tracker has become one of the most preferred tracking filters because of the high computational efficiency and robustness to missed detections or occlusions, especially for linear and Gaussian systems. The KF-based VOT method uses the background subtraction method for foreground object detection through background modeling and foreground segmentation and updates the object’s state recursively by model prediction and measurement correlation.

For the linear time-invariant (LTI) system model with the LTI measurement system, KF is proven to be the optimal linear minimum variance estimator [46]. However, in practice, the model mismatches and inaccurate noise statistics result in sub-optimal filtering that outputs unbiased and even divergent estimates [47]. The adaptive KF is one effective approach and has been applied in many applications [48,49,50]. In the KF-based object tracking task, the prior knowledge of the process and measurement noise covariance matrices is hard to determine due to occlusions and changes in object appearance. Therefore, designing adaptive KF focusing on noise covariance estimation is of vital importance.

Existing noise covariance estimation methods typically utilize historical open- loop data, which can be categorized as Bayesian [51], maximum likelihood estimation (MLE) [52,53], covariance matching (CM) [54,55], and subspace identification (SI) [56], correlation [57,58,59,60,61,62]. Each of these methods has its strengths and weaknesses. For instance, while the Bayesian method provides optimal posterior estimation using Bayes’ rule to update the prior knowledge of the noise covariance, it suffers from computational complexity and sensitivity to hyper-parameter choices [51]. Similarly, while the MLE method maximizes the likelihood of innovations without prior knowledge, it is computationally intensive and sensitive to model misspecification [53]. The CM method, although robust and computationally efficient, is sensitive to outliers and unsuitable for online scenarios [63]. The correlation method, and especially the auto-covariance least-squares (ALS) method has attracted quite considerable attention as it provides an unbiased estimate based on the autocovariance of the innovations with acceptable computational complexity and no specific noise model requirements [60].

Therefore, the autocovariance least square estimation (ALS) method is integrated into the KF-based VOT method, and the AKF-ALS algorithm is proposed to address the requirements of simplicity and robustness in real-time visual object tracking under uncertain noise covariance. Our contributions are threefold:

• The proposed AKF-ALS algorithm provides a robust and efficient solution for noise covariance estimation in the visual object tracking problem.
• A novel adaptive thresholding method based on the estimated process noise covariance that can predict sudden variations without heavy computations is proposed to improve the robustness of the BS method.
• The experiments on tracking the centroid of a moving ball subjected to projectile motion, free-fall bouncing motion, and back-and-forth linear motion are conducted to show the efficiency and superiority of the proposed AKF-ALS algorithm.

The rest of the paper is organized as follows. Section 2 formulates the VOT problem and introduces the KF and the ALS method. In Section 3, the AKF-ALS algorithm is developed to deal with the visual object tracking task with uncertain noise covariance. It utilizes the ALS method to achieve real-time noise covariance tuning and proposes a novel adaptive thresholding method to improve the robustness of the BS method. The computation complexity of the AKF-ALS algorithm is derived and a numerical analysis is conducted to show its efficiency. In Section 4, the proposed AKF-ALS algorithm is validated to outperform the Kalman filter given constant noise covariance through the experiments on tracking the centroid of a moving ball subjected to projectile motion, free-fall bouncing motion, and back-and-forth linear motion. Section 5 provides concluding remarks and a discussion of future work.

2. Problem Formulation and Preliminaries

The visual object tracking problem consists of two main stages: background subtraction for object detection and Kalman filter for object tracking.

2.1. Background Subtraction

Background subtraction involves distinguishing the foreground (moving objects) from the background in video frames. We can formulate this problem mathematically as follows. The input video sequence is represented as a series of frames: $F_1,F_2,\cdots ,F_t,\cdots ,F_N$, where N is the total number of frames and t is the current time instance. Each frame $F_t$ is a 2D array of pixels denoted as $F_t(i,j)$, where i and j represent the spatial coordinates of the pixel in the frame. Denote the input image at time t as $I(x,y,t)$ which is composed of a static background $B(x,y)$ and a moving foreground $F(x,y,t)$, such that:

$$I(x,y,t)=B(x,y)+F(x,y,t)$$

(1)

The goal of background subtraction is to create a binary mask $M(x,y)$ for each frame $F_t(i,j)$, where pixels belonging to moving objects are labeled as 1 (foreground), and the rest are labeled as 0 (background).

$$M(x,y)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}|I(x,y,t_d)-I(x,y,t_f)|>T_t\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(2)

where $T_t$ is the user-defined threshold, $t_d$ and $t_f$ represent the time instances of the dark and bright frames, respectively.

2.2. Kalman Filter

Kalman filter is a recursive estimator that provides the optimal solution under the assumptions of a linear system and Gaussian noise. The Kalman filter operates in a two-step process recursively: prediction and correlation. In the prediction step, the filter predicts the state and error covariance estimates to obtain the prior estimates for the next time step.

$$\begin{array}{cc}\hfill {\widehat{x}}_{k+1}& =F{\widehat{x}}_k+B{u}_k\hfill \\ \hfill P_{k+1}& =F{P}_k{F}^T+Q\hfill \end{array}$$

(3)

where ${\widehat{x}}_k$ is the state estimate at time k, $P_k$ is the state covariance matrix, and Q is the process noise covariance.

In the correction step, the filter incorporates a new measurement into the prior estimate to obtain an improved posterior estimate.

$$\begin{array}{cc}\hfill K_{k+1}& =P_{k+1}{H}^T{(H{P}_{k+1}{H}^T+R)}^{-1}\hfill \\ \hfill {\widehat{x}}_{k+1}& ={\widehat{x}}_{k+1}+K_{k+1}(z_{k+1}-H{\widehat{x}}_{k+1})\hfill \\ \hfill P_{k+1}& =(I-K_{k+1}H)P_{k+1}\hfill \end{array}$$

(4)

where $K_{k+1}$ is the Kalman gain, R is the measurement noise covariance.

2.3. Auto-Covariance Least-Squares (ALS) Method

The auto-covariance least-squares (ALS) method is introduced to estimate the unknown noise covariance in a linear state-space model in model-based estimation and control, which is crucial for improving the performance of a visual object tracking task.

The ALS method is based on the innovation sequence, denoted as $e_k$, which is defined as the difference between the observed measurement and the estimated measurement based on the predicted state in the prediction step:

$$e_k=z_k-H{\widehat{x}}_{k|k-1},\forall k.$$

(5)

Denote the residual, which is the difference between the actual state and the predicted state as ${\epsilon }_k=x_k-{\widehat{x}}_{k|k-1}$. Then the residual and the innovation follow $e_k=H{\epsilon }_k+v_k$, and

$${\epsilon }_{k+1}=\underset{\overline{F}}{\underbrace{(F-FKH)}}{\epsilon }_k+\underset{G}{\underbrace{[I_{n_x},-FK]}}\underset{{\overline{w}}_k}{\underbrace{\left[\begin{array}{c}{w}_k\\ v_k\end{array}\right]}}.$$

(6)

In the steady state, the Kalman gain converges to a constant value, which can be computed offline. The steady-state residual covariance matrix, denoted by $P_{ϵ}$, can be solved via the following Lyapunov equation:

$$P_{\epsilon }=\overline{F}{P}_{\epsilon }{\overline{F}}^{\mathrm{T}}+G\Sigma G^{\mathrm{T}}$$

(7)

where $\Sigma =\left[\begin{array}{cc}Q& 0_{n_x\times n_z}\\ 0_{n_z\times n_x}& R\end{array}\right]$.

Then the auto-covariance of the innovation sequence is:

$$\begin{array}{cc}\hfill C_{e,0}& \triangleq \mathbb{E}\left[e_k{e}_k^{\mathrm{T}}\right]=H{P}_{\epsilon }{H}^{\mathrm{T}}+R\hfill \\ \hfill C_{e,j}& \triangleq \mathbb{E}\left[e_{k+j}{e}_k^{\mathrm{T}}\right]=H{\overline{F}}^j{P}_{ϵ}{H}^{\mathrm{T}}-H{\overline{F}}^{j-1}FKR\hfill \end{array}$$

(8)

where $j=1,2,\dots ,N-1$, and N is a user-defined parameter defining the window size. Substitute the solution to (7) into (8), and the noise covariance matrix is derived as a linear matrix equation by using the least-squares method:

$$\mathcal{A}\theta =b$$

(9)

where the matrix $\mathcal{A}$ is defined as

$$\begin{array}{cc}\hfill \mathcal{A}& =\left[D,D(FK\otimes FK)+(I_{n_z}\otimes \Gamma )\right]\hfill \\ \hfill D& =\left(H\otimes \mathcal{O}\right){\left(I_{n_x^2}-\overline{F}\otimes \overline{F}\right)}^{-1}\hfill \\ \hfill \mathcal{O}& ={\left[H^{\mathrm{T}},{\left(H\overline{F}\right)}^{\mathrm{T}},\dots ,\left(H{\overline{F}}^{N-1}\right)\right]}^{\mathrm{T}}\hfill \\ \hfill \Gamma & ={\left[I_{n_z},-{\left(HFK\right)}^{\mathrm{T}},\dots ,-{\left(H{\overline{F}}^{N-2}FK\right)}^{\mathrm{T}}\right]}^{\mathrm{T}}\hfill \end{array}$$

(10)

where the symbol ⊗ stands for the Kronecker product. Denote the dependent variables as $\theta ={[Q_s^{\mathrm{T}},R_s^{\mathrm{T}}]}^{\mathrm{T}}$ and $b={\left(C_e\left(N\right)\right)}_s$ with $C_e\left(N\right)={[C_{e,0},C_{e,1}^{\mathrm{T}},\dots ,C_{e,N-1}^{\mathrm{T}}]}^{\mathrm{T}}$. $Q_s$ means the column-wise stacking of the matrix Q into a vector.

The ALS method estimates the covariance matrices by minimizing the difference between the sample and theoretical innovations’ autocovariance by minimizing the following cost function:

$$\begin{array}{cc}\hfill \widehat{\theta }& =arg\underset{\theta }{min}{\parallel \mathcal{A}\theta -b\parallel }_2^2\hfill \\ \hfill & s.t.,Q,R_i\ge 0.\hfill \end{array}$$

(11)

Then the parameter $\theta$ is estimated by solving the semi-definite constrained least squares problem. The optimal solution can be found by setting the derivative of the cost function to zero when the matrix inequality holds:

$$\widehat{\theta }={\left({\mathcal{A}}^{\mathrm{T}}\mathcal{A}\right)}^{-1}{\mathcal{A}}^{\mathrm{T}}\widehat{b}={\mathcal{A}}^{†}\widehat{b}$$

(12)

where $\widehat{b}$ denotes the unbiased estimate of the vector b, and is computed by using the ergodic property of the N-innovations to estimate the auto-covariance matrix $C_e\left(N\right)$:

$$\begin{array}{cc}\hfill \widehat{b}& ={\left({\widehat{C}}_e\left(N\right)\right)}_s\hfill \\ \hfill {\widehat{C}}_e\left(N\right)& ={[{\widehat{C}}_{e,0}^{\mathrm{T}},{\widehat{C}}_{e,1}^{\mathrm{T}},\dots ,{\widehat{C}}_{e,N-1}^{\mathrm{T}}]}^{\mathrm{T}}\hfill \\ \hfill {\widehat{C}}_{e,j}& =\frac{1}{\tau -j}\sum _{k=1}^{\tau -j}{e}_{k+j}{e}_k^{\mathrm{T}},j=0,1,\dots ,N-1\hfill \end{array}$$

(13)

where $\tau$ is denoted as the window size of the auto-covariances.

The steady-state solution $P_{\epsilon }$ to the Lyapunov equation in (7) exists and the innovation sequence is a stationary process if the prediction error transitional matrix $\overline{F}$ is stable.

$$P_{\epsilon }=F(P_{\epsilon }-P_{\epsilon }{H}^{\mathrm{T}}{(H{P}_{\epsilon }{H}^{\mathrm{T}}+R_A)}^{-1}H{P}_{\epsilon })+Q_A$$

(14)

where $Q_A$ and $R_A$ are positive user-defined matrices.

3. ALS-KF Based Visual Object Tracking Algorithm

The visual object tracking algorithm based on the Kalman filter combined with the autocovariance least square estimation method called the AKF-ALS algorithm is proposed in this paper. It consists of three procedures: object detection via background subtraction, object tracking using the Kalman filter, and noise covariance estimation using the ALS method, as shown in the flowchart in Figure 1.

3.1. Object Detection Using Background Subtraction

The first step in the tracking process is to identify the object of interest. In our proposed method, we utilize the background subtraction method to detect the moving object. The background is initialized as the first frame of the video. For each subsequent frame, we subtract the background from the current frame. By applying a threshold to the difference image, we convert it into a binary image. The binary image is then processed using morphological operations to reduce noise and fill gaps in the detected objects.

Object tracking in video sequences often requires adaptive thresholding techniques to robustly handle changes in object appearance over time. The process noise covariance $\widehat{Q}$ estimated by the Kalman filter provides a principled metric to adapt threshold levels based on the dynamics of the tracked object. The process noise captures uncertainties in modeling the motion of the object, thus larger values of $\widehat{Q}$ imply more aggressive maneuvers by the target. This leads to greater shape deformation and scale variation as the object moves rapidly through the scene. By leveraging $\widehat{Q}$, the threshold T can be adapted during tracking according to:

$$T_t=\alpha {\widehat{Q}}_t+T_{min}$$

(15)

where $\alpha$ is a scaling parameter and $T_{min}$ is a minimum threshold level. The key idea is that a larger process noise covariance $\widehat{Q}t$ at time t indicates faster motion, which requires relaxing the threshold to accommodate greater changes in appearance. This avoids losing the target due to overly tight thresholds based on stale measurements.

Connected components in the resulting binary image are then identified, and a bounding box is generated for each component. The centroid of each component, which is computed as the geometric center of the bounding box, is then stored as the object’s location. The process is repeated for each frame in the video, resulting in a series of centroids representing the object’s trajectory.

3.2. Object Tracking Using Kalman Filter

Once the objects are detected via the background subtraction, the next step is to track the detected object using the Kalman filter. Denote the object $O_i$ is represented by its state $x_i$, which includes its position and velocity. In the visual object tracking task, the Kalman filter is used to predict the state of each object in the next frame, given its current state and the observations (detected positions). The Kalman filter is a recursive algorithm that uses a series of measurements observed over time, containing statistical noise and other inaccuracies, and produces estimates of unknown variables that tend to be more precise than those based on a single measurement alone. It includes the prediction and update steps: The prediction step projects forward the current state and error covariance estimates to obtain the a priori estimates for the next time step; the update step incorporates a new measurement into the a priori estimate to obtain an improved a posteriori estimate.

Given the current state of the object $O_i$ at time t denoted as $x_{i,t}$, and observation $z_t$, we want to estimate the state $x_{i,t+1}$ at time $t+1$. The observation $z_t$ can be seen as the detected position of the object from the background subtraction. The dynamic model of the system and the measurement model can be represented as:

$$\begin{array}{cc}\hfill x_{k+1}& =F{x}_k+B{u}_k+w_k\hfill \\ \hfill z_k& =H{x}_k+v_k\hfill \end{array}$$

(16)

where $x_k$ is the state vector, $u_k$ is the control input, $w_k$ is the process noise, $z_k$ is the measurement, and $v_k$ is the measurement noise. F, B, and H are system matrices.

Based on the Kalman filter introduced in (3) and (4); the KF derivation for the object tracking problem is given below.

Prediction:

$$\begin{array}{cc}\hfill {\widehat{x}}_{k|k-1}& =F{\widehat{x}}_{k-1|k-1}+B{u}_k\hfill \\ \hfill P_{k|k-1}& =F{P}_{k-1|k-1}{F}^T+Q\hfill \end{array}$$

(17)

Correction:

$$\begin{array}{cc}\hfill K_k& =P_{k|k-1}{H}^T{(H{P}_{k|k-1}{H}^T+R)}^{-1}\hfill \\ \hfill {\widehat{x}}_{k|k}& ={\widehat{x}}_{k|k-1}+K_k(z_k-H{\widehat{x}}_{k|k-1})\hfill \\ \hfill P_{k|k}& =(I-K_{k}H)P_{k|k-1}\hfill \end{array}$$

(18)

3.3. Noise Covariance Estimation Using ALS

An important part of the Kalman filter is the estimation of the noise covariance matrices Q (process noise covariance) and R (measurement noise covariance). The accuracy of these estimates significantly affects the filter’s performance. In this work, we use the ALS method to estimate Q and R. The ALS method is based on the autocovariance of the innovation sequence (also known as the difference between the actual measurement and the predicted measurement based on the predicted state) from the Kalman filter. The ALS method computes the autocovariance of this sequence and formulates a least-squares problem to solve for Q and R. By continuously updating the noise covariance estimates, the ALS method allows the Kalman filter to adapt to changes in the system dynamics and measurement noise, resulting in improved tracking accuracy.

The adaptive Kalman filter (AKF) with the autocovariance least square estimation (ALS) algorithm, termed AKF-ALS algorithm is outlined in Algorithm 1.

Algorithm 1 AKF-ALS

1: Initialize:  ${\widehat{x}}_0,P_0,F,H,Q,R$   2: for $k=1,2,\cdots$ until convergence do   3:     Prediction:   4:     ${\widehat{x}}_{k|k-1}=F{\widehat{x}}_{k-1|k-1}$   5:     $P_{k|k-1}=F{P}_{k-1|k-1}{F}^{\mathrm{T}}+Q$   6:     Update:   7:     $e_k=z_k-H{\widehat{x}}_{k|k-1}$   8:     $K_k=P_{k|k-1}{H}^{\mathrm{T}}{(H{P}_{k|k-1}{H}^{\mathrm{T}}+R)}^{-1}$   9:     ${\widehat{x}}_{k|k}={\widehat{x}}_{k|k-1}+K_k{e}_k$ 10:     $P_{k|k}=(I-K_{k}H)P_{k|k-1}$ 11:     if $k\ge \tau +N$ then 12:         Estimate $Q,R$ via ALS: 13:         Compute ${\widehat{C}}_{e,0},{\widehat{C}}_{e,1},\cdots ,{\widehat{C}}_{e,N-1}$ in Equation (13) 14:         Stack ${\widehat{C}}_{e,0},{\widehat{C}}_{e,1},\cdots ,{\widehat{C}}_{e,N-1}$ into $\widehat{b}$ 15:         Compute $\mathcal{A}$ using Equation (10) 16:         Solve for $\widehat{\theta }$ using Equation (12) 17:         Unstack $\widehat{\theta }$ into $\widehat{Q}$ and $\widehat{R}$ 18:     end if 19: end for

The algorithm includes the following procedures.

• Initialize the state estimate ${\widehat{x}}_0$, the state covariance matrix $P_0$, the system matrices F and H, the Kalman gain matrix K, the window sizes N and $\tau$, and the noise covariance matrices Q and R.
• The main loop of the algorithm iterates until the ALS method has converged. At each iteration, The state estimate and covariance matrix are updated recursively using the observed measurement $z_k$, the measurement model defined by H, and the noise covariance matrix R.
• After a sufficient number of iterations k exceeds $\tau +N$, the ALS method can be applied to estimate the noise covariance matrices. We compute the autocovariance of the innovations, stack them into a vector $\widehat{b}$, and compute the matrix $\mathcal{A}$ as per the provided equations. A least-squares problem is then solved to get an estimate $\widehat{\theta }$, which we then unstack into the individual matrices $\widehat{Q}$ and $\widehat{R}$.

Process model owing to its computational simplicity and efficiency, which is essential for adherence to real-time tracking constraints. The LTI model’s predictive strength in capturing linear motion makes it particularly suitable for VOT scenarios, and its compatibility with Kalman filtering allows for optimal tracking in systems characterized by Gaussian noise and linear dynamics. Concurrently, our measurement model is robustly designed to withstand VOT-specific challenges such as occlusions and dynamic appearance changes, employing autocovariance least square estimation (ALS) to dynamically adapt to the non-stationary noise statistics typical of real-world environments. This adaptability is augmented by an adaptive background subtraction pre-processing step that effectively differentiates the target from its backdrop, thus enhancing measurement accuracy and, by extension, the reliability of object tracking.

The computation complexity of the proposed AKF-ALS algorithm is derived to show the real-time performance in Theorem 1.

Theorem 1

(Computation complexity). The computational complexity of the proposed AKF-ALS algorithm in (1) is $O\left(N^2{n}_x^4{n}_z^4\right)$, where $n_x$ and $n_z$ are the dimensions of the state and measurement, respectively.

Proof. 

In (10), the computation of ${\overline{F}}_{n_x\times n_x}$ matrix involves $O\left(n_x^2\right)$ for matrix addition, $O\left(n_x^3\right)$ for matrix multiplication and $O\left(n_x^3\right)$ for solving the Lyapunov equation in (7). Computing $\overline{F}\otimes \overline{F}$ involves $O\left(n_x^3\right)$ for fast Kronecker product computation, then the inverse matrix ${\left(I_{n_x^2}-\overline{F}\otimes \overline{F}\right)}_{n_x^2\times n_x^2}$ involves $O\left(n_x^6\right)$ using the fastest matrix inversion method. ${\mathcal{O}}_{N{n}_z\times n_x}$ and the fourth term ${\Gamma }_{N{n}_z\times n_z}$ involve concatenating matrices, with a complexity of $O\left(N{n}_x^2{n}_z\right)$ and $O\left(N{n}_x^2{n}_z\right)$, respectively. ${\mathcal{A}}_{N{n}_z^2\times (n_x^2+n_z^2)}$ has a complexity of approximately $O\left(max(n_x^6,N{n}_x^4{n}_z^2)\right)$. Then, the ALS estimate $\widehat{\theta }$ involves $O\left(N^2{n}_z^4{n}_x^4\right)$ for computing ${\mathcal{A}}_i^{\mathrm{T}}{\mathcal{A}}_i$ and $O\left(n_x^6\right)$ for the matrix inversion ${\left({\mathcal{A}}_i^{\mathrm{T}}{\mathcal{A}}_i\right)}^{-1}$. Given the computation complexity of KF is $O\left((max(n_x^3,n_z^3)\right)$, the proposed AKF-ALS algorithm has the computation complexity of $O\left(max(n_x^6,N^2{n}_x^4{n}_z^4,(\tau -j)N{n}_z^2)\right)$. Since $n_x$ and $n_z$ are near and $(\tau -j)N$ is smaller than $n_x^4{n}_z^2$ in most cases, the computation complexity is streamlined into $O\left(N^2{n}_x^4{n}_z^4\right)$. □

As indicated by the above theorem, incorporating the ALS into the KF does not produce a significant rise in the computation complexity when $n_x$ and $n_z$ are small.

4. Experiment Study

4.1. Numerical Analysis of Noise Covariance Estimation

A numerical simulation of noise covariance estimation for the LTI system with the LTI measurement model in (16) is provided to show the performance of the ALS algorithm, where the sampling time, the state, and measurement transitional matrices are set as $T_s=1$, $F=\left[\begin{array}{cc}0.7T_s\hfill & 0.3\hfill \\ -0.1\hfill & 0.9T_s\hfill \end{array}\right]$ and $H=[0.5,0.2]$.

In this simulation study, the ALS algorithm was operated with $N_{MC}=500$ Monte Carlo simulations over $N_{time}=150$ time steps. The initial state for each simulation was defined as $x_0={[1.1,1.1,1.1,1.1]}^{\mathrm{T}}$, expressed in meters (m). The true noise covariance matrices were initialized with $Q_{true}=\left[\begin{array}{cc}0.5& 0\\ 0& 1\end{array}\right]$ and $R_{true}=0.8$, where the units are square meters ($m^2$). The initial estimations for the noise covariance matrices were set to $Q_A=0.1I_4$ for the process noise and $R_A=I_2$ for the measurement noise.

The noise covariance estimates ${\widehat{Q}}_{ALS}$, ${\widehat{R}}_{ALS}$ is estimated using the ALS method. Then the steady-state state covariance is computed by solving the Lyapunov equation in (7) as $P_{\epsilon }=\left[\begin{array}{cc}0.32\hfill & 0.14\hfill \\ 0.14\hfill & 0.35\hfill \end{array}\right]$, the steady-state Kalman gain is $K=[0.17,0.13]$.

By setting the window size $N_w$ as 5, The empirical estimation of N-innovations autocovariance is computed as $\widehat{b}={[2.019,1.144,1.177,0.861,0.794]}^{\mathrm{T}}$. Then according to the Equation (12), the estimated noise covariance matrix with 500 MC initialization is computed and stored as ${\widehat{Q}}_{ALS,MC}$ and ${\widehat{R}}_{ALS,MC}$, and the mean value of ${\widehat{Q}}_{ALS,MC}$ and ${\widehat{R}}_{ALS,MC}$ is ${\widehat{Q}}_{ALS}=\left[\begin{array}{cc}0.48& 0.214\\ 0.214& 0.879\end{array}\right]$ and ${\widehat{R}}_{ALS}=0.801$.

The effectiveness of the estimated noise covariance matrices, based on 500 Monte Carlo simulations, is depicted in Figure 2 and Figure 3. Figure 2 plots the diagonal elements of the estimated model noise covariance matrix, ${\widehat{Q}}_{11}$ and ${\widehat{Q}}_{22}$, on the x-axis and y-axis, respectively. Figure 3 presents the estimated measurement noise covariance matrix R on the x-axis, with the density of the Monte Carlo simulations displayed on the y-axis.

As indicated above, the noise covariance estimates with 500 MC initialization follow the Gaussian distribution where the mean value $Q_{ALS}$ and $R_{ALS}$ are near the true value. The results demonstrate that the ALS method can accurately estimate the noise covariance for the KF, and the ALS-KF framework which combines the Kalman filter with the ALS method is well-suited for real-world visual tracking applications where the underlying noise distributions are unknown.

4.2. Centroid-Based Moving Ball Tracking Subjected to Three Motions

The experiment on the video of a moving ball separately subjected to projectile motion, free-fall bouncing motion, and back-and-forth linear motion across the video is conducted to assess the performance of the proposed AKF-ALS algorithm. The video consists of 100 frames, with a resolution of 480 × 640 pixels.

• Projectile motion following a parabolic path refers to the motion of an object thrown into the air and subject to downward acceleration due to gravity. The challenge in tracking such motion is the constantly changing speed and direction of the object.
• Free-fall bouncing motion is the motion of an object falling under gravity and then bouncing back upwards. The challenge is the object’s velocity changes rapidly at the point of impact, which can be difficult for a tracking algorithm to handle.
• Back-and-forth linear motion refers to an object moving to and fro along a straight line. The challenge is the abrupt change in velocity when the object changes direction.

For visual object tracking (VOT), these different types of motion pose unique challenges. Rapid changes in direction and velocity may affect the ability of a VOT system to accurately track an object. Therefore, the proposed AKF-ALS algorithm is applied to show the performance. In the algorithm, the background subtraction method is first applied to each frame in the video to detect the ball. The background is initialized as the first frame of the video, and an adaptive threshold is applied to the difference image to create a binary image. The connected components in the binary image are identified as potential objects, and the centroid of the largest component is stored as the location of the ball. Then the Kalman filter is used to track the ball’s movement, i.e., the centroid of a series of video frames. The ALS method is used to estimate the process and measurement noise covariance. The performance of the proposed method was compared with a standard Kalman filter with fixed noise covariance.

Specifically, we consider the state of the ball as its position and velocity, and the observation as the detected position of the ball from the background subtraction. Based on the kinematic equation, the relation between the position x and velocity $\dot{x}$ can be written as follows:

$${\mathrm{x}}_k=\left[\begin{array}{c}{x}_k\hfill \\ {\dot{x}}_k\hfill \end{array}\right]=\left[\begin{array}{c}{x}_{k-1}+{\dot{x}}_{k-1}\Delta t+1/2{\ddot{x}}_{k-1}\Delta t^2\\ {\dot{x}}_{k-1}+{\ddot{x}}_{k-1}\Delta t\end{array}\right]$$

(19)

where $\Delta t$ is the sampling time and can be set as $0.2$. The 2-D kinetic equation can be shown for state ${\mathrm{x}}_k$ as:

$${\mathrm{x}}_k=\left[\begin{array}{c}{x}_k\\ y_k\\ {\dot{x}}_k\\ {\dot{y}}_k\end{array}\right]=\left[\begin{array}{c}{x}_{k-1}+{\dot{x}}_{k-1}\Delta t+1/2{\ddot{x}}_{k-1}\Delta t^2\\ y_{k-1}+{\dot{y}}_{k-1}\Delta t+1/2{\ddot{y}}_{k-1}\Delta t^2\\ {\dot{x}}_{k-1}+{\ddot{x}}_{k-1}\Delta t\\ {\dot{y}}_{k-1}+{\ddot{y}}_{k-1}\Delta t\end{array}\right]$$

(20)

which can be simplified as:

$${\mathrm{x}}_k=\left[\begin{array}{cccc}1& 0& \Delta t& 0\\ 0& 1& 0& \Delta t\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]{\mathrm{x}}_{k-1}+\left[\begin{array}{cc}1/2\Delta t^2& 0\\ 0& 1/2\Delta t^2\\ \Delta t& 0\\ 0& \Delta t\end{array}\right]a_{k-1}$$

(21)

where the transitional matrices F, $\Gamma$ and H are defined as

$$\begin{array}{cc}\hfill & \mathrm{F}=\left[\begin{array}{cccc}1& 0& \Delta t& 0\\ 0& 1& 0& \Delta t\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]\hfill \\ \hfill & \Gamma =\left[\begin{array}{cc}1/2\Delta t^2& 0\\ 0& 1/2\Delta t^2\\ \Delta t& 0\\ 0& \Delta t\end{array}\right]\hfill \\ \hfill & \mathrm{H}=\left[\begin{array}{cccc}1\hfill & 0\hfill & 0\hfill & 0\hfill \\ 0\hfill & 1\hfill & 0\hfill & 0\hfill \end{array}\right].\hfill \end{array}$$

(22)

The performance of the proposed AKF-ALS-based VOT algorithm is compared to that of the traditional KF-based VOT algorithm, as introduced in [64], across three types of motion: projectile motion, free-fall with a bouncing trajectory, and linear back-and-forth motion. The KF-based VOT algorithm uses the same kinematic equation in (19), but the noise covariance is initially determined as a constant value, i.e., $Q=1$ and $R=0.5$. The comparative results are illustrated in Figure 4.

As indicated from the above figure, the proposed AKF-ALS algorithm shows improved robustness in handling constantly changing speed and direction, rapidly changing velocity, and the abrupt change in velocity when the object changes direction, while continuing tracking accurately after the change ends. As the tracking progresses, the performance of KF-based VOT approaches that of AKF-ALS-based VOT because the state covariance matrix approximates the steady-state value. This highlights the advantage of the proposed AKF-ALS-based VOT method in handling challenging conditions in visual object tracking.

The performance of the proposed ALS-KF-based VOT algorithm is evaluated using the root mean square error (RMSE) (in pixels) between the estimated position ${\widehat{x}}_{1:2,t}$ (the first two elements) and the true position $x_{1:2,t}$ of the moving ball subjected to projectile motion, free-fall bouncing motion, and back-and-forth linear motion across the video, as shown in

Table 1. Comparisons of RMSE (in pixels) between AKF-ALS-based and KF-based VOT subjected to projectile motion, free-fall bouncing motion, and back-and-forth linear motion.

RMSE (in Pixels)	Projectile	Free-Fall Bouncing	Back-and-Forth Linear
AKF-ALS	1.05	2.76	0.89
KF	4.67	8.35	2.31

.

$$RMSE=\sqrt{\frac{1}{N}\sum _{t=1}^N{(x_{1:2,t}-{\widehat{x}}_{1:2,t})}^2}$$

(23)

The quantitative analysis of the table above reveals that AKF-ALS consistently achieves lower root mean square error (RMSE) values across various motion patterns. The RMSE for AKF-ALS is 1.05 pixels for projectile motion, significantly better than the 4.67 pixels of the standard KF. For free-fall bouncing motion, AKF-ALS records an RMSE of 2.76 pixels, in contrast to 8.35 pixels for the KF. Additionally, in back-and-forth linear motion scenarios, AKF-ALS demonstrates an RMSE of 0.89 pixels, outperforming the KF’s 2.31 pixels. The results show that the proposed AKF-ALS algorithm achieves a lower RMSE compared to a standard KF VOT algorithm with fixed noise covariance, demonstrating the proposed method outperformed the standard Kalman filter in terms of RMSE.

4.3. Bounding Box-Based Pedestrian Tracking with Occlusions

The simulation was also conducted on the PETS09-S2L1 dataset [65] within the MOT-15 challenge, a complex scenario with up to eight pedestrians engaging in unusual patterns. The IOU matching utilized the detection box from the previous frame to compute the IOU with the current frame detections. A threshold of $IO{U}_{\mathrm{Threshold}}=0.3$ was set for a match to be considered valid. If the IOU matching failed to find a maximum value due to occlusion, the Kalman filter’s prediction was used to estimate the target’s new position and continue tracking. However, the KF-based VOT algorithm failed to maintain tracking under occlusion conditions.

The AKF-ALS VOT algorithm demonstrated robustness in maintaining the identity of the pedestrian target even when the IOU matching failed due to occlusion due to its dynamic covariance estimation, which allows the filter to adapt its parameters in response to the evolving state of the system. This adaptive mechanism is crucial for handling occlusions and erratic pedestrian movements. The AKF-ALS VOT algorithm improves upon the standard KF VOT by dynamically adjusting the process and measurement noise covariances. The state vector remains consistent with the classic AKF, with the following representation:

$$X={[x,y,h,w,\dot{x},\dot{y}]}^T$$

(24)

where x and y denote the horizontal and vertical position of the pedestrian in the video frame (in pixels), respectively, and $\dot{x}$ and $\dot{y}$ denote the horizontal and vertical velocity of the pedestrian (in pixels per frame). h and w denote the height and width of the bounding box around the pedestrian (in pixels), respectively.

The ALS algorithm updates the process noise covariance matrix Q and the observation noise covariance matrix R iteratively. The innovation sequence is used to calculate the autocovariance, which in turn adjusts Q and R to respond to abrupt changes in the tracked object’s motion. The trajectory is shown in Figure 5.

As indicated by the figure, the tracking of pedestrians in scenarios obstructed by objects such as a light pole, can present challenges for bounding box detection methods. However, the implementation of the adaptive Kalman filter with adaptive least squares (AKF-ALS) algorithm demonstrates robustness in tracking despite failures in bounding box detection under such conditions.

To demonstrate the comparative performance of the accuracy and robustness, we use the RMSE in (23), the identity switches, and the track fragmentation as the metrics. The identity switches metric measures the number of times the tracking algorithm incorrectly switches the identity of the tracked object. Fewer identity switches indicate a more robust tracking performance. The track fragmentation evaluates the continuity of the tracking throughout the sequence. Lower numbers imply fewer instances where tracks are broken and later reacquired. We compare the proposed ALS-AKF algorithm with the classic adaptive KF algorithm using Otsu’s method in [66], as shown in

Table 2. Comparisons of RMSE (in pixels), between AKF-ALS-based and KF-based VOT algorithm in pedestrian tracking with occlusions.

Metric	AKF ALS VOT	Classic AKF VOT [66]
RMSE	5.5 pixels	10.7 pixels
Identity switches	1	8
Track fragmentation	2	8

.

As is indicated from the table above, the AKF-ALS VOT algorithm demonstrated a markedly higher precision with a root mean square error (RMSE) of only 5.5 pixels, compared to the classic AKF VOT’s 10.7 pixels. Robustness in maintaining pedestrian identity was also significantly improved in the AKF-ALS VOT, which had a mere single identity switch, whereas the classic AKF VOT recorded eight. Additionally, the AKF-ALS VOT showed greater continuity in tracking with only two instances of track fragmentation, in contrast to the classic AKF VOT’s eight, indicating a more stable tracking performance throughout the sequences. These enhancements are likely due to the AKF-ALS’s iterative updates to the process and observation noise covariance matrices, enabling it to adeptly adjust to sudden changes in the tracked object’s motion.

5. Conclusions

In conclusion, the AKF-ALS algorithm introduced in this paper represents a significant advancement in the field of real-time visual object tracking. By integrating the autocovariance least squares estimation method with an adaptive Kalman filter, we have demonstrated a robust approach to tracking that compensates for system model mismatches and invalid offline calibration, which are critical in dynamic environments. The computational complexity analysis indicates that AKF-ALS is efficient enough for real-time applications, striking an optimal balance between computational load and tracking accuracy. Our experimental results have shown that the AKF-ALS algorithm exhibits superior performance over traditional Kalman filter methods that rely on fixed noise statistics. The algorithm’s capability to adapt to changing noise covariances in real-time ensures greater accuracy in tracking the centroid of a moving object under various motion patterns, including projectile motion, free-falling with bounces, and linear back-and-forth motion. Experimental results on the complex PETS09-S2L1 dataset from the MOT-15 challenge demonstrate that the AKF-ALS VOT algorithm is capable of tracking pedestrians with occlusion where traditional methods fail. The algorithm’s dynamic adaptation to changing noise covariances in real-time significantly enhances accuracy and reliability in tracking various motion patterns. Overall, the AKF-ALS algorithm’s ability to iteratively refine noise covariances and adjust to abrupt motion changes makes it an excellent candidate for autonomous vehicles or mobile robotics.

Future work should focus on expanding the validation of the AKF-ALS algorithm across a broader range of applications, including those with more complex and non-linear object dynamics. Additionally, further research could explore the integration of this algorithm with deep learning techniques to enhance tracking performance in highly cluttered and dynamic visual scenes.