Learning Unsupervised Cross-Domain Model for TIR Target Tracking

Abstract

The limited availability of thermal infrared (TIR) training samples leads to suboptimal target representation by convolutional feature extraction networks, which adversely impacts the accuracy of TIR target tracking methods. To address this issue, we propose an unsupervised cross-domain model (UCDT) for TIR tracking. Our approach leverages labeled training samples from the RGB domain (source domain) to train a general feature extraction network. We then employ a cross-domain model to adapt this network for effective target feature extraction in the TIR domain (target domain). This cross-domain strategy addresses the challenge of limited TIR training samples effectively. Additionally, we utilize an unsupervised learning technique to generate pseudo-labels for unlabeled training samples in the source domain, which helps overcome the limitations imposed by the scarcity of annotated training data. Extensive experiments demonstrate that our UCDT tracking method outperforms existing tracking approaches on the PTB-TIR and LSOTB-TIR benchmarks.

1. Introduction

Compared with the RGB target tracking sequences, the thermal infrared (TIR) target tracking sequences have no color information which is not affected by illumination changes. Therefore, the TIR target tracking can be applied to intelligent transportation, video surveillance, sea rescue, and other scenarios at night or in rainy/foggy weather. The scarcity of TIR training samples makes the trained feature extraction network perform an inferior target feature representation, which significantly restricts the performance of the TIR target tracking method. Although there are some methods to solve the problem of insufficient training data, such as few-shot learning [1,2], domain adaptation [3,4], domain transfer [5,6], etc., there is still a lot of room for improvement in TIR target tracking tasks.

Since training samples of RGB target tracking are plentiful and easy to obtain, RGB target tracking has already been extensively studied for several decades. There are many excellent RGB target tracking frameworks, e.g., the discriminative correlation filters (DCF) based tracking frameworks [7,8,9] and the deep learning based tracking frameworks [10,11,12,13]. The DCF-based trackers usually train a correlation filter to model the relationship of the target feature and its corresponding label [14,15,16]. The deep learning-based trackers need abundant training samples to train their convolutional feature extraction network that can represent the target more comprehensively [17,18,19]. In [20], Algabri and Choi propose a robust recognizer combined with deep learning techniques to adapt to different lighting in scene ambient lighting and employ an enhanced character recognition model online update strategy to solve the problem of target character appearance change drifting during tracking. Inspired by the success of deep learning-based RGB target tracking methods, some TIR target tracking methods are trying to improve their performance by leveraging an off-the-shelf network to extract target features [21,22,23]. The STAMT [21] tracking method leverages a pre-trained convolutional feature extraction network to extract these deep features for a better target representation. The HSSNet [22] tracker refers to the Siamese architecture [24,25,26,27] which views the tracking task as similarity template matching and uses the Siamese-based convolutional neural network with multi-convolutional layers to obtain the semantic and spatial features of the tracking TIR target. These TIR target trackers utilize the off-the-shelf network for feature extraction which achieved some performance improvements; however, the tracking performance of these trackers is still significantly limited because this network is not trained for the TIR target-tracking task [21,28,29]. To make the feature extraction network adapt to the TIR tracking task, it is necessary to use TIR training samples for their convolutional feature extraction network training [30,31,32]. However, the deficiency of the TIR training samples makes it a difficult problem to train such a convolutional feature extraction network on these TIR training samples [33,34].

To resolve the aforementioned drawback, we propose an unsupervised cross-domain model (UCDT) for the TIR tracking task. Although these TIR training samples are insufficient, the RGB training samples are numerous and easy to obtain. Therefore, we propose a cross-domain model to transfer the convolutional feature extraction network trained on the RGB domain (source domain) to the TIR domain (target domain) so that the feature extraction network trained on abundant RGB training samples can be used for target representation in the TIR domain. Considering that such abundant RGB training sample annotation in the RGB domain is very time-consuming and expensive, we adopt an unsupervised learning-based method to generate pseudo labels for these unlabeled training samples. This unsupervised pseudo-label generation method can effectively reduce the cost of the sample annotation so that abundant training samples can be used for the feature extraction network training. The trained feature extraction network can extract the comprehensive target representation features, which can make the tracker obtain accurate tracking results. Figure 1 illustrates comparison results in some complex TIR tracking scenarios between our UCDT tracker and some trackers. The figure indicates that our UCDT tracking results are similar to the ground-truth labels of the target, it shows the effectiveness of our unsupervised cross-domain model in the TIR target-tracking task.

The following are the main contributions of this work:

• We propose an unsupervised cross-domain model for the TIR tracking task.
• The cross-domain model could transfer the convolutional feature extraction network trained in the source RGB domain to the target TIR domain for the target feature extraction, which can improve the poor representation of the trained convolutional feature extraction network effectively due to insufficient training samples in the target TIR domain.
• To make the trained convolutional feature extraction network have a strong target representation capability, an unsupervised learning method is adopted to generate pseudo-labels for the unlabeled training samples in the source RGB domain for the convolutional feature extraction network training.
• Extensive experimental results show the competitiveness of the proposed tracking method compared with other tracking methods on the PTB-TIR [35] and the LSOTB-TIR [36] benchmarks.

2. Related Works

Deep Thermal Infrared Tracking: Deep learning has been widely used in image processing tasks because of its powerful representation ability [29,37,38]. Nowadays, deep learning-based trackers have attracted more and more researchers due to their good representation ability [39,40,41]. Ma et al. [39] present a coarse-to-fine tracking model by combining multi-level convolutional features within the DCF-based framework which can receive more accurate results than DCF-based trackers that only use hand-crafted features. The TADT [40] tracker uses two auxiliary tasks to select target-aware features online to achieve a more compact match between the template and the candidate. In [41], Gundogdu et al. propose a deep convolutional feature based on a classifier combined with the DCF-based framework for the TIR tracking task. Liu et al. [42] present an integrated TIR tracer using a pre-trained CNN model to extract multiple convolution features of the tracking target for a relatively complete representation of the target. To mitigate the crisis caused by this problem, the MMNet [28] tracking method proposes a multi-task framework for the TIR domain tracking task. Despite these trackers having obtained some TIR target tracking results, there is still a big room for improvement in the TIR tracker for its tracking performance compared to the real scenario requirements.

Domain Gap of RGB Images and TIR Images: Deep learning-based trackers require a large amount of labeled data for their model training [43,44,45]. The RGB source domain has relatively sufficient training data, while the TIR target domain has very limited data [31,46,47,48]. Many TIR target trackers use pre-trained RGB-based feature extraction networks, but TIR images capture temperature information through thermal infrared radiation, which is different from RGB’s visible light [21,32,33]. This difference leads to a domain gap, making it challenging to transfer models from RGB to TIR. RGB images are sensitive to lighting changes, while TIR images work well in varying light conditions, including complete darkness. Consequently, applying RGB-trained models to TIR data can lead to performance degradation due to differences in feature distribution [28,29,49]. To bridge this gap, domain adaptation techniques are needed to adapt RGB models to TIR features or retrain models specifically for TIR [50,51]. The limited TIR training samples further restrict tracking accuracy. Utilizing an unsupervised cross-domain model to transfer RGB-trained convolutional networks to TIR can significantly improve tracking accuracy in the TIR domain.

Unsupervised Feature Representation: The target feature representation is an objective way of quantifying targets of interest. Typically, researchers use these hand-crafted features (e.g., HOG, SIFT) to measure targets of interest in the image. By extracting a sequence of various features from the target image, it can be represented as a collection of values. These feature representations can be applied to a variety of tasks, such as image segmentation [52,53], or object classification [54,55]. Due to the defects of these hand-crafted features, visual tasks based on these hand-crafted features can not achieve ideal performance. In most computer vision tasks, convolutional neural networks (CNN) have obtained the best results [56,57]. CNN-learned features are thought to be more sensitive to meaningful visual material than hand-crafted features, making them a viable choice for feature-based image analysis. However, using CNN to learn features is very challenging due to it being heavily dependent on the training tasks [58,59]. An unsupervised approach that leads to high-quality feature learning without the label of training samples will solve the bottleneck of time-consuming and expensive training sample labeling [60,61,62,63,64]. In [60], an unsupervised learning-based UDT tracker has been proposed to reduce the need for annotated training samples in the model training process. Sun et al. [61] present uses a cross-modal distillation method to extract the representation of TIR modes from RGB modes on abundant unannotated paired RGB-TIR samples, which could effectively improve the tracking accuracy of the TIR tracker. Wang et al. [63] propose to use tracking as a proxy task for computer vision systems to learn the visual representation of video-related tasks. The above study utilized various pretexts constructed in the video, including time-sequence verification [60], displacement prediction [61] and background discrimination [64]. In this work, we mainly use multi-cycle consistency loss to perform forward-backward tracking in the deep DCF-based tracking framework to generate pseudo labels of these unlabeled training samples in the source RGB domain for the network training, which can reduce the requirement of sample annotation while ensuring the representation ability of the trained convolutional feature extraction network.

3. The Proposed UCDT Tracker

We propose an unsupervised cross-domain model (UCDT) based tracker for more accurate TIR target tracking accuracy. First, we briefly introduce the deep correlation tracking framework, which includes a deep convolutional neural network for target feature extraction and discriminative correlation filters for the target response map generation. Next, we propose a cross-domain model to transfer the feature extraction network trained in the source RGB domain into the target TIR domain for the target feature extraction, which could improve the poor target representation in the TIR domain. To reduce the cost of labeling these training samples, we adopt an unsupervised learning method to generate pseudo-labels for the unlabeled training samples in the source RGB domain to train the convolutional feature extraction network.

3.1. Deep Correlation Tracking Framework

The deep correlation tracking framework includes a deep CNN model for target feature extraction and a discriminative correlation filter (DCF) layer for the target response map generation. Figure 2 shows the TIR target tracking process under the deep correlation tracking framework. In the first frame, the target of interest is given as the template, and the CNN is used to extract features and train the parameters of the DCF layer.

The DCF-based tracker trains a classifier to separate the target of interest from backgrounds [9,14]. The DCF layer can be expected as:

$$W=\underset{W}{argmin}\left(\right||W\ast X-{Y\left|\right|}^2+{\lambda \left|\right|W\left|\right|}^2),$$

(1)

where W is the trained parameters of the DCF layer in the deep correlation tracking framework, X is training samples, Y is the corresponding labels, and $\lambda$ is a regularization parameter. By using the Fourier transform, the convolution operation in Equation (1) is transformed into the form of dot multiplication, which reduces the computational complexity and improves the tracking speed. The parameters W of the DCF layer can be achieved as follows:

$$W={\mathcal{F}}^{-1}\left({\displaystyle \frac{\widehat{X}\odot \widehat{Y}}{{\widehat{X}}^{\ast }\odot \widehat{X}+\lambda }}\right).$$

(2)

where $\mathcal{F}$ is the Fourier transform, $\widehat{X}$ is the training samples in the Fourier domain, $\widehat{X}$ is the corresponding labels in the Fourier domain, and $\lambda$ is a regularization parameter. In the subsequent search frames, the CNN is used to extract features of the search area S, and convolved with the trained parameters to obtain the response map R:

$$R={\mathcal{F}}^{-1}(\mathcal{F}\left(S\right)\odot \mathcal{F}\left(W\right)).$$

(3)

The center position of the target is determined by finding the position of the maximum point in the response map, and then the tracking result on the search frame can be obtained by using a scale estimation strategy [16,18].

3.2. Cross-Domain Model

The cross-domain model focuses on improving model performance in the target domain by leveraging labeled data from the source domain. We give the following definition: (1) Source Domain $D_s$ includes feature space ${\mathcal{X}}_s$ and label space ${\mathcal{Y}}_s$, along with the sample set $S={\left\{(x_i^s,y_i^s)\right\}}_{i=1}^{n_s}$. In our unsupervised cross-domain model, the labels of the samples in the source domain are generated pseudo-labels, which are generated in a manner described in Section 3.3. (2) Target Domain $D_t$ includes feature space ${\mathcal{X}}_t$ and label space ${\mathcal{Y}}_t$, along with the sample set $T={\left\{\left(x_i^t\right)\right\}}_{i=1}^{n_t}$, where samples in the target domain are unlabeled. The sample distributions of the source domain and target domain are typically different, i.e., $P_s(x,y)\ne P_t(x,y)$, where $P_s(x,y)$ represents the joint distribution in the source domain, and $P_t(x,y)$ represents the joint distribution in the target domain. Mathematically, we aim to minimize the distribution difference between the source and target domains to achieve domain adaptation. Due to the shortage of training samples that can be directly used in the TIR target tracking task, we propose a cross-domain model to train a feature extraction network using the training samples in the RGB domain, and the trained network could be transferred to the TIR domain for target feature extraction. This operation enables the training of the convolutional feature extraction network in the RGB source domain which could be deployed for the TIR target domain. The network architecture consists of two shared convolutional layers, and each convolution layer is followed by a normalization layer to prevent model over-fitting and multiple task-specific layers. The shared layers mainly share some network parameters, while the specific layer has different parameters for different tasks.

After careful consideration, these convolutional layers and normalization layers are selected as the feature extraction layers that are shared by the RGB domain and the TIR domain. The DCF-based layer is considered the task-specific layer. The overall of our proposed unsupervised cross-domain model is illustrated in Figure 3. The parameters of feature extraction networks in different domains are shared, but the parameters of classifiers are not shared. Although it is difficult to find the roles for each layer in the CNN architecture semantically, we can infer that these two CNN layers are considered feature encoders. They encode input samples in the learned common feature space. The DCF layer uses these features to carry out classification tasks. If multiple domains need to be encoded in a common feature space, and the characteristics of the domains are significantly different, different encoders are required for each domain. In the proposed cross-domain model, we technically create different DCF layers for different domains, which can be conducted by simply sharing layers across different encoders, but only setting up the DCF layer for different domains.

3.3. Unsupervised Pseudo-Label Generation

The training process of the convolutional feature extraction network requires abundant labeled training samples. The training samples in the RGB domain are very rich but the labeled training samples are very limited. A large number of unlabeled samples cannot be directly applied to network training and sample annotation is time-consuming and expensive. Therefore, we adopt an unsupervised learning way to generate pseudo labels for large-scale unlabeled samples, which are then used in the training of the convolutional feature extraction network to ensure the target representation ability of the trained network and reduce the demand for network training on labeled samples. As shown in Figure 4, it is the unsupervised pseudo-label generation process for the unlabeled training samples.

In general, robust tracking methods can guarantee the consistency of forward-backward tracking results. The accuracy of generated pseudo labels of the unlabeled training samples can be improved by using the forward-backward tracking consistency. As shown in Figure 5, we use a multi-cycle consistency method to optimize and improve the generated pseudo labels of the unlabeled training samples. The distances of forward and backward tracking results can be represented by the following loss function:

$$l=\left|\right|R_t-R_t^{\prime }{\left|\right|}^2+\left|\right|R_{t+k}-R_{t+k}^{\prime }{\left|\right|}^2+\left|\right|R_{t+2k}-R_{t+2k}^{\prime }{\left|\right|}^2,$$

(4)

where l is the training loss, $R_t$, $R_{t+k}$, $R_{t+2k}$ represent the forward tracking response map on the corresponding frame, and $R_t^{\prime }$, $R_{t+k}^{\prime }$, $R_{t+2k}^{\prime }$ represent the backward tracking response map on the corresponding frame. In this paper, k is set to 2. The accuracy of the generated pseudo labels of the unlabeled training samples is optimized by continuously reducing the distances of forward and backward tracking results on the same frame.

4. Experiments

To verify the performance of the proposed unsupervised cross-domain tracking method (UCDT), we made some comparative experiments for the UCDT and several state-of-the-art tracking methods on PTB-TIR [35] and LSOTB-TIR [36] benchmark datasets. Following the PTB-TIR [35] benchmark, we used the precision and success scores as the evaluation metrics to evaluate the proposed UCDT tracker. More details of the testing benchmarks and evaluation metrics can be found in [35,36].

4.1. Implementation Details

The experiments were implemented in Matlab2021b with MatConvNet toolbox on a PC with an i9-11900KF 3.50 GHz CPU, 64 GB RAM, and an NVIDIA GeForce RTX 3080Ti GPU. The tracking speed is approximately 29 fps. In the unsupervised cross-domain model trained by the SGD with a momentum of $0.9$ the weight decay is set to $5\times {10}^{-4}$. The network is trained for 30 epochs with a mini-batch size of 32. The learning rate exponentially decays from $1\times {10}^{-2}$ to $1\times {10}^{-5}$. The regularization parameter $\lambda$ is set to $0.01$, the scale factor $\alpha$ is set to $1.02$, the trend of the target scale $\sigma =-1,0,1$, and the model update parameter $\eta$ is set to $0.025$. The sizes of the feature extraction CNN layers are $3\times 3\times 3\times 32$ and $3\times 3\times 32\times 32$, respectively. A local response normalization layer (relu) is employed at the end of each CNN layer to reduce the gradient calculation and avoid over-fitting.

4.2. Experiments on PTB-TIR Benchmark

We compared our UCDT tracking method with some other trackers, which including STAMT [21], TADT [40], MMNet [28], MCCT [65], Staple [66], UDT [60], DSiam [27], CREST [18], MLSSNet [67], MCFTS [42], SianFC [24], HSSNet [22], SiamTri [68], HDT [10], CFNet [69], and HCF [39] on the PTB-TIR [35] benchmark. Figure 6 shows the experimental results of these comparison tracking methods. From Figure 6 we could see that our UCDT tracker achieved the best success score and the second-best precision score. Compared to these deep DCF-based tracking methods [10,18,39] that use the off-the-shelf convolutional neural networks for target feature extraction, our UCDT tracking method uses an unsupervised cross-domain model for feature extraction which obtains improvements in tracking accuracy obviously. Although the precision score of our UCDT tracking method is lower than the STAMT [21] tracking method, our UCDT tracking method is higher than the success score, which shows that the proposed UCDT tracking method is competitive compared to the STAMT [21] tracking method. The proposed unsupervised cross-domain model is effective in TIR target tracking scenarios, as demonstrated by these experimental tracking results.

Figure 7 draws the comparison of our UCDT tracker and other trackers for the background clutter, scale variation, occlusion, and out-of-view attributes on the PTB-TIR [35] benchmark in the success metric. It was obvious that the proposed UCDT tracker outperformed other trackers when it came to background clutter, scale variation, and occlusion. To more comprehensively evaluate the tracking accuracy of different tracking methods, we present a comparison result of the success rate and tracking speed for different trackers on the PTB-TIR [35] benchmark. The comparison result is shown in Figure 8, which clearly shows that our UCDT tracker achieves a good balance in tracking success rate and tracking speed.

4.3. Experiments on LSOTB-TIR Benchmark

To further evaluate the tracking accuracy of our UCDT tracker, we made some comparisons of it with some other trackers (including TADT [40], SiamMask [26], STAMT [21], BACF [14], SRDCF [16], UDT [60], MCCT [65], SianFC [24], SiamTri [68], CREST [18], Staple [66], MCFTS [42], MLSSNet [67], CFNet [69], HSSNet [22], HCF [39], HDT [10] and DSiam [27]) on LSOTB-TIR [36] benchmark. Figure 9 shows these comparison results of the mentioned tracking methods. We know that the proposed UCDT tracking method achieved the best scores in precision, normalized precision, and success metrics (72.6%, 64.8%, 60.1%). Compared to these Siamese-based tracking methods (e.g., SiamMask [26], SiamFC [24], SiamTri [68]), our UCDT tracking method performs better in each evaluation metric. Compared to these DCF-based tracking methods (e.g., BACF [14], SRDCF [16], HCF [39]), our UCDT tracking method has demonstrated strong advantages in tracking performance. Compared to these convolutional neural network learning-based tracking methods (e.g., CREST [18], TADT [40], HDT [10]), our UCDT tracking method benefits from the unsupervised cross-domain model which obtains more accurate tracking performance. These comparative results demonstrate the efficiency of our UCDT tracking method in a variety of ways.

The comparisons of other tracking methods and our UCDT tracking method on some different attributes are shown in Figure 10. The comparison result shows that our UCDT tracking method obtained the best tracking accuracy on these attributes, which shows the validity of the unsupervised cross-domain model in the proposed UCDT tracking method. What is more,

Table 1. Success scores (%) comparison of our UCDT and other trackers. The top three are highlighted in red, blue and green respectively.

Trackers	UCDT	SiamMask	STAMT	TADT	MLSSNet	MCCT	BACF	SRDCF	CREST	Staple	HSSNet	HDT	HCF	CFNet	SiamFC	SiamTri	DSiam	UDT	MCFTS
	Ours	[26]	[21]	[40]	[67]	[65]	[14]	[16]	[18]	[66]	[22]	[10]	[39]	[69]	[24]	[68]	[27]	[60]	[42]
DM	53.5	55.1	$51.8$	53.4	$44.4$	$44.9$	$49.2$	$43.8$	$47.3$	$42.0$	$36.9$	$39.6$	$43.3$	$38.4$	$53.0$	$51.6$	$35.2$	$43.1$	$45.9$
HH	56.5	$54.9$	61.8	57.5	$41.1$	$44.8$	$46.4$	$48.0$	$51.9$	$42.4$	$39.1$	$42.2$	$40.8$	$33.9$	$54.0$	$55.3$	$44.0$	$42.2$	$44.0$
VM	74.6	72.8	$66.6$	$68.6$	$54.2$	$67.4$	$70.0$	74.1	$45.2$	$66.3$	$44.8$	$35.8$	$31.4$	$58.7$	$56.2$	$52.0$	$46.9$	$71.1$	$57.9$
VS	60.1	$54.9$	$54.0$	58.0	$47.1$	55.5	$53.9$	$52.2$	$53.6$	$51.0$	$42.9$	$41.4$	$42.8$	$41.7$	$46.6$	$47.2$	$30.1$	$52.7$	$47.7$

shows that our UCDT tracking method achieved top-three success scores on four different tracking scenarios. Compared to these DCF-based trackers (e.g., BACF [14], SRDCF [16]), the proposed UCDT tracker achieves significant advantages in all four scenarios. Compared to these deep learning-based trackers (e.g., SiamMask [26], TADT [40], STAMT [21]), the proposed UCDT tracker achieves competitive tracking performance. The results in Table 1 also show that most trackers used directly for RGB on TIR tracking tasks can only obtain a suboptimal result. In conclusion, the proposed UCDT tracker produced some competitive tracking outcomes when compared to other tracking methods.

4.4. Qualitative Comparison

To visually illustrate the tracking accuracy of our UCDT tracker, we made some comparisons of UCDT with some other trackers, including UDT [60], SiamMask [26], TADT [40], and STAMT [21] on challenging thermal infrared target testing video sequences. To make the comparison results more intuitive, we give the target’s ground-truth label for comparison. Figure 11 shows the compared results on some TIR test tracking sequences in the LSOTB-TIR [36] benchmark. The target-aware-based TADT [40] tracking method easily interfered with occlusion and the distractor (e.g., deer-H-001, and person-D-019). The STAMT [21] tracker performs well on the deer-H-001 sequence, which benefits from its structural target-aware model. However, these testing results of the STAMT tracker are still unacceptable on other sequences (such as hog-H-001, street-S-002, and person-S-008). As shown in Figure 11 almost all trackers achieved good results in the tracking scenario (e.g., motobiker-V-001) which comes from the vehicle-mounted camera (VM). In other tracking scenarios which come from the surveillance camera (e.g., street-S-002, person-S-008), the drone-mounted camera (e.g., person-D-019), and the hand-held camera (e.g., deer-H-001, hog-H-001), the tracking results of other trackers all have great defects except that our UCDT tracker can obtain results similar to the target’s ground-truth label.

5. Conclusions

In this paper, an unsupervised cross-domain model has been proposed for the thermal infrared target-tracking task. The cross-domain model could effectively use the training samples in the source domain to train a general convolutional feature extraction network for target feature extraction in the target domain. Meanwhile, using the unsupervised method to generate pseudo labels for the unlabeled training samples in the source domain can effectively solve the problem that the trained convolutional feature extraction network has insufficient ability of target representation due to insufficient labeled training samples. Extensive experimental results show that our UCDT tracker has achieved some competitive tracking results. The unsupervised cross-domain model relies on the training samples in the source domain, which makes it impossible to evaluate the trained model on the ground truth label of the target domain, limiting the accuracy and optimization effect of the TIR tracking performance. In the future, we will consider using methods such as few-shot learning to further improve the TIR tracking method.