TensorTrack: Tensor Decomposition for Video Object Tracking

Abstract

Video Object Tracking (VOT) is a critical task in computer vision. While Siamese-based and Transformer-based trackers are widely used in VOT, they struggle to perform well on the OTB100 benchmark due to the lack of dedicated training sets. This challenge highlights the difficulty of effectively generalizing to unknown data. To address this issue, this paper proposes an innovative method that utilizes tensor decomposition, an underexplored concept in object-tracking research. By applying L1-norm tensor decomposition, video sequences are represented as four-mode tensors, and a real-time background subtraction algorithm is introduced, allowing for effective modeling of the target–background relationship and adaptation to environmental changes, leading to accurate and robust tracking. Additionally, the paper integrates an improved multi-kernel correlation filter into a single frame, locating and tracking the target by comparing the correlation between the target template and the input image. To further enhance localization precision and robustness, the paper also incorporates Tucker2 decomposition to integrate appearance and motion patterns, generating composite heatmaps. The method is evaluated on the OTB100 benchmark dataset, showing significant improvements in both performance and speed compared to traditional methods. Experimental results demonstrate that the proposed method achieves a 15.8% improvement in AUC and a ten-fold increase in speed compared to typical deep learning-based methods, providing an efficient and accurate real-time tracking solution, particularly in scenarios with similar target–background characteristics, high-speed motion, and limited target movement.

1. Introduction

Object tracking plays a vital role in various applications, enabling the reliable identification and localization of objects in real-world scenarios. Its importance extends across a range of fields, including but not limited to security surveillance, industrial automation, autonomous vehicles, and augmented reality. Object tracking is an essential aspect of ensuring efficient and safe operation, enhancing user experiences, and improving overall performance. With advancements in computer vision and machine-learning techniques, object tracking continues to evolve, driving the creation of new and innovative applications.

Traditional methods for object tracking have typically relied on the use of correlation filters and hand-crafted features or heuristics to locate and track objects. Correlation filters are filters that operate by comparing the correlation of an object template with the input image to locate and track the object. However, these filters are often limited in their ability to handle complex object appearances, changes in illumination, background clutter, and occlusion, making them unreliable in real-life scenarios. To address these limitations, researchers have turned to machine-learning techniques such as tensor decomposition for more robust and effective object tracking. Tensor decomposition can learn a model of the object’s appearance and adapt to changes in the environment, improving accuracy and efficiency.

Although correlation filters have gained popularity for their effectiveness and ease of implementation, tensor decomposition remains an emerging technique in the domain of robust object tracking. However, recent studies have demonstrated the versatility of tensor decomposition in different fields, including spatiotemporal hotspot detection, stress recognition [1], georeferenced data analysis [2], and low-rank tensor approximation using compressive sensing [3]. These applications showcase the potential and capabilities of tensor decomposition for solving real-world problems. While not as widely used in object tracking as correlation filters, tensor decomposition remains a promising technique for improving the accuracy and robustness of object-tracking algorithms. Therefore, further research is necessary to explore its full potential in this domain.

A video sequence can be represented as a four-mode tensor of dimensions $w\times h\times c\times b$, where w and h denote the width and height of each frame, c represents the number of color channels, and b corresponds to the frame index of the current image. Tensor decomposition aims to reveal the intrinsic structure of high-dimensional data, providing an effective mathematical framework for video data modeling and analysis. Since the target object in a video sequence is typically distinguishable from the background, video data exhibits a high degree of structural regularity, making it particularly suitable for tensor decomposition techniques. By leveraging tensor decomposition, a low-dimensional intrinsic representation of the data can be extracted, enhancing both interpretability and computational efficiency in video processing and analysis.

The majority of existing correlation filter (CF) methods in single-object tracking (SOT) focus on configuring various features within a single frame, such as those found in [4,5,6,7,8]. Some methods even incorporate deep perceptive pooling (DPP) features into the CF method, as seen in [9]. Although many of these methods have achieved state-of-the-art (SOTA) results on OTB100, they have not been able to compete with Deep Neural Networks (DNNs) in terms of processing speed.

Through extensive experimentation and evaluation, the effectiveness of the proposed method has been demonstrated, particularly in improving tracking robustness across various challenging scenarios. The integration of tensor decomposition with correlation-based tracking enhances the adaptability of the framework, allowing it to handle complex tracking conditions more effectively. By introducing a novel decomposition strategy into video object tracking, this work advances the field and lays the foundation for future research. The key contributions of this paper are as follows:

1. Developed a novel framework that integrates tensor decomposition techniques with single-object tracking methods, achieving superior performance even on platforms with limited computational resources.

2. Designed a fast and efficient algorithm based on Real-Time Tensor-Based Background Subtraction, leveraging tensor decomposition to dynamically learn the target’s appearance model and quickly adapt to changing environments.

3. Integrated appearance and motion patterns via Tucker2 decomposition, generating composite heatmaps that enhance localization precision and robustness in visually ambiguous, high-motion, and low-motion scenarios.

2. Related Work

Recent advances in object-tracking approaches have been heavily influenced by CF methods, which have been widely applied since the pioneering work by Bolme in 2010 [10]. Over the past eight years, CF methods have gained popularity SOT [4]. Various improved versions of CF algorithms have been proposed to enhance tracking performance. For instance, Tang et al. [5] developed a tracker based on Multi-Kernel Correlation Filters (MKCF), leveraging the discriminatory power spectra of different features. Lukezic and Vizivc [6] introduced a novel formulation that effectively handles visual and geometric constraints using a constellation model of CFs. Moreover, Nikouei et al. [11] proposed a lightweight hybrid tracking algorithm called Kerman, aiming to implement intelligent surveillance as an edge service. Additionally, Liu et al. [12] conducted a survey on CF algorithms for object tracking, discussing the research background and recent advances in the field and introducing major object-tracking datasets.

To address the limitations of CF-based methods in handling complex target appearances, illumination variations, background clutter, and occlusions, researchers have proposed innovative approaches. Fu et al. [7] presented a novel Object-Aware Saliency-Guided Dually Regularized CF (DRCF) that utilizes saliency maps to improve the learning and adaptation of filters, particularly in occlusion situations. Ma et al. [13] introduced Average Peak-to-Correlation Energy (APCE), a new standard for multiresolution translation filter frameworks, which aims to achieve more robust and accurate scale estimation. Furthermore, Li et al. [14] developed a tracker called ADTrack, based on Discriminative Correlation Filters (DCF), which incorporates illumination adaptation and anti-dark capability. Despite these advancements, the limited ability of CF-based methods in practical scenarios remains a challenge. Therefore, it is necessary to develop more expressive feature representations and robust optimization strategies, leading to the exploration of alternative paradigms.

Tensor decomposition into orthogonal factors is a common practice in many fields, including statistics, machine learning, and signal processing. Various tensor decomposition (TD) techniques have been proposed and applied in different scientific and engineering domains. De et al. [15] introduced a novel TD method called Constrained Factor (CONFAC) decomposition. Kolda [16] provided an overview of software and applications for higher-order TD. Grigis and Richard [17] proposed a novel High-Order (HO) TD suited for redirection. Kiraly and Sabharwal [18] studied Orthogonal Outer Product Decompositions, where factors in the addition are orthogonal between additions. Favier et al. [19] introduced a novel tensor model called Nested Tucker Decomposition (NTD), applicable to Tensor Space-Time Coding (TSTC) in MIMO relay systems. Xu et al. [20] proposed an efficient TD method for decomposing Vandermonde factor matrices for Direction of Arrival (DOA) estimation in transmit beamforming MIMO radar. Azaiez et al. [21] used TD and one-dimensional interpolation in data-driven order reduction modeling to reduce the dimensionality of high-dimensional systems. Zhao et al. [22] introduced a statistical method named SSR-Tensor that detects temporal consistency hotspots effectively in spatiotemporal datasets through TD. Hutter et al. [23] evaluated the performance of low-rank TD methods in modeling application performance.

While TD is not common in SOT, Ratre et al. [24] applied it in their research. They described the application of TD in object categorization tasks, where they combined Tucker Tensor Decomposition (TTD) with Gaussian Mixture Models (GMM). Although their study focused on classification, the TTD method also holds potential for application in SOT.

TensorTrack is a tracker that combines Template Tracking and Detection with SOT. It exhibits competitive performance compared to deep neural network (DNN) methods. Furthermore, TensorTrack has been optimized to run efficiently on resource-constrained platforms, which contributes to its improved performance.

3. Preliminary

This paper aims to provide a brief introduction to CF and TTD. Using the framework of TensorTrack as an example, this section will give an overview of their fundamental principles and concepts.

3.1. Overview of Correlation Filters in Object Tracking

Correlation filters compute the similarity between a target template and input images for object localization. In their seminal work, Bolme et al. [10] proposed the Minimum Output Sum of Squared Error (MOSSE) filter, which initializes a filter using a single frame to ensure robust tracking. Subsequent studies, such as Henriques et al. [25], introduced techniques like the Fast Fourier Transform (FFT) and the Kernel Trick to enhance filter accuracyb. Danelljan et al. [26] further improved filter performance by projecting RGB images into an 11-color channel representation, while Bertinetto et al. [27] proposed a probability-based correlation filter, advancing SOT capabilities.

Figure 1 illustrates the overall framework of TensorTrack, integrating TTD within a correlation filter pipeline for robust object tracking. The process starts with the input video frame sequence, where the target is manually annotated in the first frame using a bounding box. The region of interest (ROI) containing the target is cropped from subsequent frames to focus on relevant areas, reducing extraneous data. Next, features are extracted from the cropped frames, effectively representing the target’s appearance and structural information. These extracted features are then processed using Tucker Tensor Decomposition to reduce dimensionality while preserving critical spatial and temporal information. This step significantly enhances the efficiency and robustness of the framework.

In the learning phase, the correlation filter leverages the decomposed features to construct a statistical model, capturing key patterns and relationships for accurate target tracking. This process is described by Equation (1):

$$\widehat{H_i}=\eta \ast {\displaystyle \frac{G_i⨀F_i}{F_i⨀F_i}}+(1-\eta )\ast {\widehat{H}}_{i-1}$$

(1)

Here, ${\widehat{H}}_i$ represents the learned filter at iteration i, where the hat notation ^{^} denotes an estimation, ⨀ indicates element-wise multiplication, and capital letters represent the Fourier transform. The iterative nature of this formula allows the correlation filter to adapt to changes in the target’s appearance over time, ensuring robust and reliable tracking performance across frames.

During the detection stage, the correlation filter estimates the target’s position in each subsequent frame by computing:

$$\widehat{Y_i}={\widehat{H}}_i⨀X_i$$

(2)

where $\widehat{Y_i}$ denotes the response map, and $X_i$ represents the extracted features in the current frame. The tracking result is obtained by identifying the location with the highest response value within a predefined search region.

To ensure continuous tracking accuracy, periodic updates of the correlation filter are required. These updates refine the learned model ${\widehat{H}}_i$ based on new observations, preventing model drift and adapting to appearance variations.

The score map ($\widehat{Y_i}$), which represents the Fourier transform of the response, indicates the target’s position based on the maximum response value, as depicted in Figure 2.

3.2. Principles of Tucker Tensor Decomposition

TTD is a mathematical approach used for analyzing multidimensional data. It breaks down a higher-dimensional tensor into lower-dimensional tensors, reducing the complexity of data representation. The Tucker decomposition identifies essential features and relationships within the data by factoring a tensor into a core tensor and factor matrices. This study employs an online Tucker2 decomposition, a specific type of tensor decomposition. In general, tensor decomposition can be classified into two types: CANDECOMP/PARAFAC (CP) tensor decomposition and Tucker tensor decomposition [28].

In general notation, a corresponds to a scalar, $\mathbf{a}$ represents a vector, A symbolizes a matrix, and $\mathcal{A}$ denotes a tensor. It should be noted that while a tensor is indexed using more than two index numbers, an n-mode tensor is indexed using n independent indices to represent its elements.

Tucker decomposition was originally proposed by Tucker in 1963 [29] and later refined in subsequent articles by Levin [30] and Tucker [28,31]. Among these works, the 1966 article by Tucker [28] is widely regarded as the most comprehensive and frequently cited reference.

Tensor-Train Decomposition reduces the rank of a tensor, explores hierarchical structures within a specific task, and minimizes redundancy in matrix representation [32,33]. The general formulation of TTD for a three-mode tensor is provided in Equation (3), where the tensor $\mathcal{X}$ is approximated by decomposing it into smaller core tensors and factor matrices using the i-mode product:

$$\mathcal{X}\approx \mathcal{G}{\times }_{1}A{\times }_{2}B{\times }_{3}C$$

(3)

where the tensors $\mathcal{X}\in {\mathbb{R}}^{I_1{I}_2{I}_3}$ and $\mathcal{G}\in {\mathbb{R}}^{I_4{I}_5{I}_6}$ are considered, along with matrices $A\in {\mathbb{R}}^{I_1{I}_4},B\in {\mathbb{R}}^{I_2{I}_5}$, and $C\in {\mathbb{R}}^{I_3{I}_6}$. The goal is to decompose the tensor $\mathcal{X}$ into $\mathcal{G}$ by utilizing the i-mode product. Specifically, for this decomposition, it is assumed that $I_4<I_1$, $I_5<I_2$, and $I_6<I_3$, as shown in Figure 3.

$${\left(\mathcal{G}{\times }_{2}B\right)}_{i_3{j}_2{i}_5}=\sum _{i_4=1}^{N_4}{g}_{i_3{i}_4{i}_5}·b_{j_2{i}_4}$$

(4)

Equation (4) represents the mode-2 tensor product, where $\mathcal{G}$ denotes the core tensor and B is the factor matrix associated with the second mode. Here, $N_4$ corresponds to the dimensionality of the decomposed mode. The operation ${\times }_2$ refers to the mode-2 product, which projects the high-dimensional tensor $\mathcal{G}$ into a lower-dimensional subspace. This operation is a fundamental step in Tucker decomposition, facilitating the extraction of key features and reducing data redundancy.

The Tucker2 decomposition itself can be framed as an optimization problem aimed at extracting the most informative features from the data, expressed as follows:

$$\underset{\begin{array}{c}U\in {\mathbb{R}}^{H_1\times d};U^{T}U=I_d\\ \mathbf{v}\in {\mathbb{R}}^{1\times d};{\mathbf{v}}^T\mathbf{v}=1\end{array}}{max}\sum _{i=1}^{W_1·3}{∥U^T{\mathbf{x}}_{\mathbf{i}}\mathbf{v}∥}_1,$$

(5)

where U represents the projection matrix that maps the input features to a lower-dimensional subspace, $\mathbf{v}$ is a unit vector, and ${\mathbf{x}}_{\mathbf{i}}$ denotes the input feature vectors. The objective of this optimization is to maximize the ${\ell }_1$ norm of the projected features while preserving the most discriminative information in the data. This ensures dimensionality reduction while maintaining essential structure in the features.

The ${\ell }_1$ norm promotes sparsity in feature selection by summing the absolute values of a vector’s elements:

$$\sum _{i=1}^N{\parallel a_i\parallel }_1=\sum _{i=1}^N\mathrm{sign}\left(a_i\right)a_i=\mathrm{sign}{\left(\mathbf{a}\right)}^T\mathbf{a}=\underset{\mathbf{b}\in {\{\pm 1\}}^N}{max}{\mathbf{b}}^T\mathbf{a},$$

(6)

where $\mathrm{sign}(·)$ denotes the sign function, which returns $\{\pm 1\}$ depending on the sign of the input. This equivalence facilitates the computation of the ${\ell }_1$ norm by reformulating it as an optimization problem involving binary variables.

It can be further shown that:

$$\sum _{i=1}^{W_1·3}{∥U^T{\mathbf{x}}_i\mathbf{v}∥}_1=\underset{\mathbf{b}\in {\{\pm 1\}}^{W_1·3}}{max}{U}^T\left(\sum _{i=1}^{W_1·3}{b}_i{\mathbf{x}}_i\right)\mathbf{v},$$

(7)

where $\mathbf{b}=[b_1,b_2,\dots ,b_{W_1·3}]$, and $b_i=\mathrm{sign}\left(U^T{x}_{i}v\right)$. This equation demonstrates how the ${\ell }_1$ norm of the projected features can be maximized by optimizing the binary weights $\mathbf{b}$. The formulation provides an efficient means of enhancing the feature representation by weighting the input vectors.

Finally, the optimization problem can be reformulated as:

$$\underset{\begin{array}{c}U\in {\mathbb{R}}^{H_{1\times d}};U^{T}U=I_d\\ \mathbf{v}\in {\mathbb{R}}^{1\times d};{\mathbf{v}}^T\mathbf{v}=1\end{array}}{max}{∥U^T{\mathbf{x}}_i\mathbf{v}∥}_1={\sigma }_{max}\left(\sum _{i=1}^{W_1·3}{b}_i{\mathbf{x}}_i\right),$$

(8)

where ${\sigma }_{max}(·)$ returns the largest singular value of the input matrix. This reformulation indicates that the Tucker decomposition process can be efficiently solved as a singular value problem, enabling effective dimensionality reduction and feature extraction.

To construct the input matrix for the optimization, the feature vectors ${\mathbf{x}}_i$ are concatenated to form $X=[{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_{W_1·3}]\in {\mathbb{R}}^{H_1\times W_1·3}$. The weighted sum ${\sum }_{i=1}^{W_1·3}{b}_i{\mathbf{x}}_i$ can then be expressed in matrix form as $X(\mathbf{b}\otimes I_M)$, where ⊗ denotes the Kronecker product. Thus, the final optimization can be written as:

$$\begin{array}{c}\hfill \underset{\begin{array}{c}U\in {\mathbb{R}}^{H_1\times d};U^{T}U=I_d\\ \mathbf{v}\in {\mathbb{R}}^{1\times d};{\mathbf{v}}^T\mathbf{v}=1\end{array}}{max}\sum _{i=1}^{W_1·3}{∥U^T{\mathbf{x}}_{\mathbf{i}}\mathbf{v}∥}_1\\ \hfill =\underset{\mathbf{b}\in {\{\pm 1\}}^{W_1·3}}{max}{\sigma }_{max}\left(X\left(\mathbf{b}\otimes I_M\right)\right).\end{array}$$

(9)

This final optimization problem highlights the efficient use of singular value decomposition in Tucker decomposition, providing a computationally feasible solution for handling high-dimensional data in resource-constrained settings.

4. Proposed Method

4.1. Video Decomposition Process

The video decomposition process is a crucial preprocessing step in TensorTrack, enabling efficient and robust tracking via Tucker Tensor Decomposition (TTD). As shown in Figure 4, video frames are converted to grayscale to reduce complexity while preserving structural information. The grayscale frames are cropped to isolate the ROI, focusing on the target and minimizing background noise. Tucker2 decomposition is then applied to extract key features and compress the data into a lower-dimensional representation. The resulting tensors are flattened for subsequent analysis, enhancing computational efficiency and forming the basis for accurate tracking in TensorTrack.

4.2. Tucker2 Decomposition and Motion Pattern

The Tucker2 decomposition extends the conventional Tucker framework by incorporating the concatenation of (n-1)-mode tensors along each axis of the original n-mode tensor, a configuration that mirrors the complex data arrangements prevalent in computer vision (CV) tasks. This advanced decomposition is succinctly expressed as:

$$\mathcal{X}=\mathcal{G}{\times }_{1}A{\times }_{2}B{\times }_{3}C,$$

(10)

where $\mathcal{X}$ denotes the multidimensional video data tensor subject to decomposition, $\mathcal{G}$ represents the core tensor, and A, B, and C are the factor matrices corresponding to the respective tensor modes. This methodology facilitates an efficient restructuring of voluminous video data, enabling a more analytical and structured approach reflective of segmentation practices within CV disciplines.

The adaptation of this framework significantly mitigates the challenges posed by data outliers, a prevalent issue identified within both Tucker and Tucker2 architectures, as documented by the literature. It also suggests a shift from the conventional L2-norm to a more robust L1-norm, a recommendation predicated on empirical evidence by Markopoulos et al. [34], which showcases an enhanced tracking accuracy within SOT frameworks employing correlation filters.

Departing from the conventional SOT methodologies, predominantly centered around feature extraction and decision fusion, this study pioneers a motion-oriented video tensor decomposition strategy. Drawing inspiration from the seminal work by Markopoulos et al. [34], the proposed method assumes a natural segregation between the target’s motion and the static backdrop, implying that the background typically manifests a low-rank tensor structure, while dynamic entities, including the target, contribute significantly to the higher-rank components. Based on this theoretical underpinning, the study advocates for the bifurcation of video clips into distinct segments of high and low rank, predominantly encapsulating the moving target within the high-rank portion. In light of the minimal spatial variance across color channels, the initial conversion of image data to grayscale is recommended, thus reducing computational load while preserving essential structural detail. This scholarly exposition articulates a nuanced equilibrium between computational efficiency and the accurate delineation of dynamic versus static elements within video streams, advancing the field of object tracking through a meticulously crafted Tucker2 decomposition strategy. Considering the tensor decomposition is an np-hard problem. The exact solution to the L1 Tucker2 is applied:

$$\underset{\begin{array}{c}U\in {\mathbb{R}}^{H_{1}xd};U^{T}U=I_d\\ \mathbf{v}\in {\mathbb{R}}^{1xd};{\mathbf{v}}^{\mathbf{T}}\mathbf{v}=\mathbf{1}\end{array}}{max}\sum _1^{W_1\ast 3}{\parallel U^T{\mathbf{x}}_{\mathbf{i}}\mathbf{v}\parallel }_{\mathbf{1}}$$

(11)

This paper proposes selecting $d=1$ to project the tensor $\mathcal{X}\in R^{(H_1,1,W_1\ast 3)}$ into a lower-rank space using the optimal matrices U and $\mathbf{v}$. In this approach, the width (W) and color (C) dimensions are combined into channels to maintain spatial consistency and ensure shared information across all channels. The vector ${\mathbf{x}}_{\mathbf{i}}\in {\mathbf{R}}^{{\mathbf{H}}_{\mathbf{1}}\times \mathbf{1}}$. Decomposing the video tensor in this way preserves critical motion information, facilitating robust tracking.

To solve for the exact solution to the formula in Equation (11), the optimization problem is reformulated as follows:

$$\underset{\mathbf{b}\in {\{\pm \mathbf{1}\}}^{{\mathbf{W}}_{\mathbf{1}}\ast \mathbf{3}}}{max}{\sigma }_{max}(X\mathbf{b}\otimes {\mathbf{I}}_{\mathbf{M}}),$$

(12)

where $X=[{\mathbf{x}}_{\mathbf{1}},{\mathbf{x}}_{\mathbf{2}},...,{\mathbf{x}}_{{\mathbf{W}}_{\mathbf{1}}\ast \mathbf{3}}]\in {\mathbf{R}}^{{\mathbf{H}}_{\mathbf{1}}\times {\mathbf{W}}_{\mathbf{1}}\ast \mathbf{3}}$, ⊗ denotes the Kronecker product, and $\mathbf{b}$ is a binary vector of length $W_1\ast 3$ with entries in $\{\pm 1\}$.

The optimal solution to Equation (12), denoted as $b_{opt}$, is then used to reconstruct the background using the following equation:

$$B_i=U_{opt}{U}_{opt}^T{X}_i{v}_{opt}{v}_{opt}^T,$$

(13)

where $U_{opt}$ and $v_{opt}^T$ are obtained via singular value decomposition (SVD) of $(X{b}_{opt}\otimes I_M)$.

4.3. Randomized Singular Value Decomposition for Efficiency

To improve computational efficiency, a randomized singular value decomposition (SVD) approach is employed. This method approximates the SVD of the original matrix using a random projection matrix, significantly reducing computational complexity while maintaining high accuracy. By leveraging randomized sketching, the proposed approach efficiently scales to large matrices, making it well-suited for real-time video processing. The corresponding Python-style pseudocode is presented in Algorithm 1.

Algorithm 1 Randomized singular value decomposition

Input: $A\in R^{m\times n}$ (target matrix), rank parameter k, power iteration parameter p

Output: $U\in R^{m\times k}$, $S\in R^{k\times k}$, $V\in R^{n\times k}$

Step 1: Randomized Initialization

$m,n=A.shape$

$\Omega =\mathrm{np}.\mathrm{random}.\mathrm{randn}(n,k)$

$Y=A\times \Omega$

Step 2: Power Iterations (Optional)

for i in range(p):

$Y=A\times (A^T\times Y)$

Step 3: QR Decomposition

$Q,R=\mathrm{np}.\mathrm{linalg}.\mathrm{qr}\left(Y\right)$

Step 4: Projection onto Lower-Dimensional Subspace

$B=Q^T\times A$

Step 5: Compute SVD on the Smaller Matrix

$\widehat{U},S,V=\mathrm{np}.\mathrm{linalg}.\mathrm{svd}(B,\mathrm{full}\_\mathrm{matrices}=\mathrm{False})$

Step 6: Map $\widehat{U}$ Back to the Original Space

$U=Q\times \widehat{U}$

Step 7: Return the Decomposition

return $U,S,V$

The computational complexity of the traditional full SVD is $O\left(m{n}^2\right)$ or $O\left(m^{2}n\right)$, making it infeasible for large-scale video sequences due to excessive computational demands. In contrast, the proposed randomized SVD reduces the complexity to $O(mnk+k^3)$, where k is the rank parameter. This reduction is achieved by first projecting the original matrix onto a lower-dimensional subspace via a randomized sketching matrix, followed by performing SVD on the reduced matrix. The significantly lower computational overhead ensures that the method remains scalable and practical for real-time tracking applications.

Despite recent advancements in single-object tracking (SOT), conventional pipelines often fail to exploit motion dynamics effectively. For instance, Bertinetto et al. [27] proposed a statistical appearance model that relies primarily on visual features while overlooking temporal information. Similarly, Zhu et al. [35] introduced object detection datasets to extract high-level features, yet their approach exhibited limited capacity for temporal modeling. Danelljan et al. [36] integrated the IoU-Net to refine target state estimation, but their framework predominantly focused on appearance-based cues. In contrast, this study introduces a tensor-decomposition-based framework that leverages Tucker2 decomposition to extract and isolate dynamic motion components from the video tensor. This approach effectively addresses critical challenges such as tracking in visually ambiguous environments where the target shares similar colors with the background and cases of rapid motion.

4.4. Integrating Appearance and Motion

When appearance models and motion patterns are fused, they can complement each other and provide richer information for inferring the position and motion of the target. Appearance models are typically based on the texture and appearance features of the target, which help determine its appearance and shape. Motion patterns, on the other hand, focus on the movement characteristics of the target, such as speed, direction, and acceleration. By combining these two models, it is possible to consider both appearance and motion information, resulting in more accurate heatmaps. A more compact heatmap means that the peaks of the heatmap are more concentrated and focused, without obvious deviation or diffusion. This compactness enables tracking algorithms to more accurately locate the target and track its movement. Compared to scattered heatmaps, compact heatmaps provide clearer and more reliable target position information, which helps improve the accuracy and robustness of target-tracking algorithms. The principle behind the motion module is pretty simple yet effective. Suppose the estimation of the appearance model to be $\widehat{A}=A+n_A$, and the estimation of the motion pattern to be $\widehat{M}=M+n_M$, where $n_A$ and $n_M$ denote the estimation errors. Assume $n_A\sim N(0,{\sigma }_A)$ and $n_M\sim N(0,{\sigma }_M)$. Then, it follows that:

$$P\left(MotionModule\right)=P\left(\widehat{A}\right)\ast p\left(\widehat{M}\right)\Rightarrow MotionModule\sim N(0,{\displaystyle \frac{{\sigma }_A^2\ast {\sigma }_M^2}{{\sigma }_A^2+{\sigma }_M^2}})$$

(14)

The above formula indicates that the composite estimation outperforms the original estimation by having a smaller variance. Furthermore, this algorithm demonstrates excellent performance in processing speed in real-time scenarios. Real-time capability is crucial in target-tracking tasks as it requires timely identification and tracking of the target to meet the demands of real-time applications.

To efficiently implement this fusion, an adaptive algorithm dynamically balances appearance and motion contributions based on their uncertainties, as outlined in Algorithm 2:

Algorithm 2 Integration of appearance and motion for heatmap fusion

Input: $A\in R^{m\times n}$, $M\in R^{m\times n}$

Output: $H_{fused}\in R^{m\times n}$Step 1: Compute Standard Deviation for Weighting

${\sigma }_A=\mathrm{np}.\mathrm{std}\left(A\right)$

${\sigma }_M=\mathrm{np}.\mathrm{std}\left(M\right)$

Step 2: Normalize Heatmaps for Consistency

$A_{norm}=A/\mathrm{np}.\max \left(A\right)$

$M_{norm}=M/\mathrm{np}.\max \left(M\right)$

Step 3: Compute Weighting Factors

$w_A={\sigma }_M^2/({\sigma }_A^2+{\sigma }_M^2)$

$w_M={\sigma }_A^2/({\sigma }_A^2+{\sigma }_M^2)$

Step 4: Compute Fused Heatmap

$H_{fused}=w_A\ast A_{norm}+w_M\ast M_{norm}$

Step 5: Normalize the Fused Heatmap

$H_{fused}=H_{fused}/\mathrm{np}.\max \left(H_{fused}\right)$

Step 6: Return Fused Heatmap

return $H_{fused}$

Figure 5 illustrates the effect of integrating appearance models and motion patterns within a compact motion module. This fusion generates a more focused and compact heatmap, enhancing its ability to accurately capture the target’s position and motion. As a result, the proposed approach improves tracking robustness and precision, particularly in dynamic and visually challenging environments.

5. Experiments

5.1. OTB100 Dataset

OTB100 is a widely used benchmark dataset for object tracking, consisting of 100 video sequences covering 22 object categories, including humans, animals, vehicles, faces, sports, and electronic devices. These sequences exhibit significant variations in appearance, scale, deformation, and occlusion, providing a diverse and challenging evaluation framework for tracking algorithms. OTB100 defines 11 tracking attributes to assess tracking performance under different challenges. The average video resolution is 356 × 530, with sequence lengths ranging from 71 to 3872 frames. Regarding background differentiation, OTB100 includes both simple and cluttered backgrounds. Some sequences feature low-contrast targets, increasing the difficulty for appearance-based trackers, while others present dynamic backgrounds and camera motion, requiring enhanced adaptability from tracking algorithms.

5.2. Implementation Details

All experiments were conducted on a workstation equipped with an Intel Core i9-10900K CPU and 32GB of RAM. To evaluate computational efficiency across different hardware settings, we conducted experiments on two different GPUs: an Nvidia RTX 1060 (6GB VRAM) and an Nvidia RTX 4070 (12GB VRAM). The proposed tracking framework and all benchmarked methods were implemented in Python using PyTorch (version 1.13) and NumPy for numerical operations. To ensure consistency, classical CF trackers and deep learning-based Siamese trackers were re-implemented based on their official source codes. To guarantee fair comparisons, all correlation filter-based trackers and our proposed method were executed on the RTX 1060 GPU. However, deep learning-based trackers (e.g., SiamR-CNN, OSTrack, TransT) exhibited significantly higher computational demands and were unable to run efficiently on RTX 1060 due to VRAM limitations. Therefore, these models were evaluated on the RTX 4070 GPU to ensure proper execution.

5.3. Impact of Tucker2 Decomposition

The proposed framework significantly enhances tracking performance on OTB100 (

Table 1. Performance and efficiency comparison on the OTB100 dataset.

Method	Precision	AUC	FPS
CF (Baseline)	0.484	0.699	42
CF+Motion Model	0.507	0.729	39
CF+Appearance Model	0.581	0.784	32
CF+Appearance Model+Tucker2 (Ours)	0.709	0.812	48

), achieving a 46.4% precision gain and a 16.2% AUC improvement over the CF baseline, demonstrating robustness in fast motion and backgrounds with high visual similarity to the target. Compared to the CF+Appearance Model, it further improves precision by 22.0% and AUC by 3.6%, highlighting the Tucker2 motion model’s role in refining accuracy and mitigating drift. These results validate the effectiveness of integrating motion and appearance cues for more reliable target localization.

To assess heatmap quality, we evaluated peak compactness and information entropy (

Table 2. Analysis of heatmap quality for different methods.

Method	Peak Compactness	Information Entropy
CF (Baseline)	-	-
CF+Motion Model	1.45	2.01
CF+Appearance Model	1.31	1.86
CF+Appearance Model+Tucker2 (Ours)	1.08	1.62

). The CF+Appearance Model+Tucker2 framework achieves the highest compactness (1.08) and lowest entropy (1.62), indicating superior noise suppression and enhanced localization, outperforming all baselines.

As illustrated in Figure 6, the first row presents the original images, while the second row displays Tucker2 decomposition outputs, effectively isolating high-rank foreground components. The third row depicts heatmaps from the appearance model, which, despite capturing target-specific features, are prone to false responses in visually similar backgrounds and exhibit susceptibility to target drift in rapid motion or occlusion scenarios. Notably, in the fifth column, where the target and background share highly similar colors, the appearance model fails to distinguish the target, resulting in a diffuse and indistinct heatmap. The fourth row represents heatmaps generated by the proposed method, which integrates motion patterns extracted via Tucker2 decomposition with appearance features. This fusion effectively mitigates challenges in scenarios with minimal relative motion, such as the Rubik clip (second-to-last column), where motion dynamics are difficult to extract due to limited target displacement. The fifth row shows the tracking results of the proposed method, which demonstrates improved target localization and robustness. By leveraging Tucker2 decomposition, the proposed approach enhances heatmap compactness and robustness, significantly improving target localization, suppressing noise, reducing false responses, and mitigating target drift, particularly in visually ambiguous or low-motion conditions.

5.4. Impact of Randomized SVD

As shown in

Table 3. Comparative experiment results of regular SVD and randomized SVD.

	Reconstruction Error (L2)	Speed (Seconds)
Regular SVD	5.45 ×  10−16	0.85
Randomized SVD	3.78 × 10−13	0.007

, randomized SVD achieves an efficiency improvement of two orders of magnitude compared to Regular SVD while maintaining a very small reconstruction error. Specifically, the speed increases from 0.85 s to 0.007 s, with only a slight increase in reconstruction error. This drastic reduction in computational time demonstrates the feasibility of randomized SVD for real-time or large-scale applications, such as high-resolution video tracking or embedded systems. While randomized SVD introduces a slightly higher reconstruction error (3.78 × 10⁻¹³ vs. 5.45 × 10⁻¹⁶), the impact on tracking performance is negligible, making it a suitable choice for scenarios with constrained computational resources.

5.5. Benchmark Evaluations

The proposed tracking framework was evaluated on the OTB100 dataset, comparing its performance and computational efficiency against classical correlation filter (CF) trackers and deep learning-based methods. As shown in

Table 4. Performance and complexity analysis on the OTB100 Dataset.

Tracker (Year)	Precision	AUC	Time Complexity	FPS	GPU Requirements
MOSSE (2010)	0.414	0.311	$O\left(N\right)$	669	RTX 1060
HCF (2015)	0.837	0.562	$O\left(N^2\right)$	10.4	RTX 1060
KYS (2020)	-	0.695	$O(N^{2}logN)$	20	RTX 1060
DR2Track (2021)	0.657	0.447	$O\left(N^3\right)$	28	RTX 1060
RCBSCF (2019)	0.711	0.485	$O\left(N^2\right)$	36	RTX 1060
Siamese Trackers	Precision	AUC	Time Complexity	FPS	GPU Requirements
SINT (2016)	0.788	0.592	$O\left(N^3\right)$	4	RTX 1060
RTINET (2019)	-	0.682	$O\left(N^3\right)$	9	RTX 1060
SiamR-CNN (2020)	0.891	0.701	$O\left(N^3\right)$	4.7	RTX 4070
Deep Learning-Based	Precision	AUC	Time Complexity	FPS	GPU Requirements
OSTrack (2022)	0.710	0.690	$O\left(N^3\right)$	35	RTX 4070
MixFormer (2022)	0.720	0.700	$O\left(N^3\right)$	30	RTX 4070
TransT (2021)	0.715	0.691	$O\left(N^3\right)$	25	RTX 4070
STARK (2021)	0.710	0.698	$O\left(N^3\right)$	20	RTX 4070
KeepTrack (2021)	0.725	0.685	$O\left(N^3\right)$	10	RTX 4070
ToMP (2022)	0.730	0.710	$O\left(N^3\right)$	20	RTX 4070
UncertaintyTrack (2024)	0.740	0.720	$O\left(N^3\right)$	15	RTX 4070
Ego-Motion (2024)	0.735	0.738	$O\left(N^3\right)$	18	RTX 4070
RaTrack (2023)	0.720	0.728	$O\left(N^3\right)$	22	RTX 4070
Tracking by 3D (2023)	0.725	0.730	$O\left(N^3\right)$	12	RTX 4070
Ours (2024)	0.709	0.812	$O\left(N^2\right)$	48	RTX 1060

, the proposed method achieves an AUC of 0.812 on the RTX 1060, demonstrating a significant 14.4% improvement over MixFormer (AUC = 0.720) in terms of tracking accuracy. While the precision of our method is slightly lower than that of some deep learning-based methods, such as SiamR-CNN (Precision = 0.891), this minor sacrifice is justified given the substantial AUC improvement, which directly contributes to tracking robustness and stability. Furthermore, compared to the CF (Baseline) method, our approach improves AUC by 16.2% and precision by 46.4%, demonstrating a clear advantage in overall tracking performance, as shown in the experimental comparison with various frameworks in Figure 7.

Regarding computational complexity, classical CF-based trackers typically exhibit linear to quadratic complexity, $O\left(N\right)$ to $O\left(N^2\right)$, allowing real-time inference but lacking robustness. In contrast, deep learning-based methods often operate with cubic complexity, $O\left(N^3\right)$, leading to substantial computational costs and a strong reliance on high-end hardware. For instance, SiamR-CNN runs at only 4.7 FPS on the RTX 4070, while even relatively efficient models like OSTrack and MixFormer only achieve 30–35 FPS, highlighting the dependency of deep learning methods on high-performance GPUs. In contrast, the proposed tracking framework maintains quadratic complexity, $O\left(N^2\right)$, enabling efficient inference on a low-end RTX 1060 GPU, reducing computational overhead by 66.7% compared to deep learning-based approaches. This significantly lowers hardware requirements, making the proposed method more accessible for deployment in real-world applications.

In terms of frame rate, the proposed method demonstrates remarkable real-time performance. On the RTX 1060, it achieves 48 FPS, representing a 60% increase over MixFormer (30 FPS) and a 37% increase over OSTrack (35 FPS), significantly surpassing deep learning-based trackers. Moreover, while classical CF trackers such as MOSSE achieve extremely high inference speeds (e.g., 669 FPS), their low accuracy limits their applicability in scenarios with high target–background similarity. The proposed framework balances accuracy and real-time performance, providing an efficient solution for object tracking, as shown in the efficiency comparison with various frameworks in Figure 8.

6. Conclusions

This paper presents TensorTrack, an innovative tracking method that integrates tensor decomposition techniques with single-object tracking, offering a promising solution for real-time tracking in resource-constrained environments. Unlike deep learning-based methods that require significant computational resources for marginal accuracy improvements, TensorTrack achieves state-of-the-art performance while maintaining computational efficiency. The method excels in scenarios involving fast motion, limited movement, and similar target–background environments, outperforming traditional methods and demonstrating competitive advantages over deep learning-based approaches, especially with significantly higher speed. Future improvements may include incorporating additional information, such as optical flow and depth data, to further enhance robustness and accuracy for more complex tracking tasks.