AD-SiamRPN: Anti-Deformation Object Tracking via an Improved Siamese Region Proposal Network on Hyperspectral Videos

Abstract

Object tracking using Hyperspectral Images (HSIs) obtains satisfactory result in distinguishing objects with similar colors. Yet, the tracking algorithm tends to fail when the target undergoes deformation. In this paper, a SiamRPN based hyperspectral tracker is proposed to deal with this problem. Firstly, a band selection method based on a genetic optimization method is designed for rapidly reducing the redundancy of information in HSIs. Specifically, three bands with highest joint entropy are selected. To solve the problem that the information of the template in the SiamRPN model decays over time, an update network is trained on the dataset from general objective tracking benchmark, which can obtain effective cumulative templates. The use of cumulative templates with spectral information makes it easier to track the deformed target. In addition, transfer learning of the pre-trained SiamRPN is designed to obtain a better model for HSIs. The experimental results show that the proposed tracker can obtain good tracking results over the entire public dataset, and that it is better than the other popular trackers when the target’s deformation is qualitatively and quantitatively compared, achieving an overall success rate of 57.5% and a deformation challenge success rate of 70.8%.

1. Introduction

Hyperspectral Object Tracking (HOT) tries to predict the position of targets continuously over Hyperspectral Videos (HSVs), using only information on the state of the targets at the initial time. For popular visual target tracking methods, it is limited in describing physical characteristics of the image, which leads to tracking failure, as the target is deformed [1,2]. Most HOT algorithms extract the spatial and spectral information for objects of interest [3], which makes it easy to distinguish objects with similar colors. In spite of the potential of hyperspectral trackers, their robustness can be negatively affected by the presence of redundant information in the HSVs [4]. Thus, there is a pressing need for further research to enhance the accuracy of tracking targets in HSVs.

Currently, there are two main tracking methods, including methods based on Deep Learning (DL) [5,6,7,8,9,10,11,12,13] and methods based on Discriminative Correlation Filtering (DCF) [14,15,16,17,18,19,20,21]. The former leverages neural networks to obtain more sophisticated depth features, which have proven to be effective for subsequent tracking. The Deep Learning Tracker (DLT) [5] was the first tracker to utilize deep learning techniques, completing the tracking process through a particle filtering framework. The feature extraction method is based on a stacked auto-encoder network. Yet, DLT tends to lose targets with partial multi-target overlap. The first application [7] of the siamese network-based tracker made by Bertinetto performs a convolution operation on the feature space of the candidate template and corresponding search region, and then determines the location of the target by finding the maximum response. The Siamese Region Proposal Network (SiamRPN) algorithm [22] draws inspiration from the principle of generating candidate regions for target detection, ultimately refining these areas to achieve a more precise bounding box. However, the algorithm’s performance can be negatively impacted when the background is mistakenly identified as the semantic object, as the SiamRPN algorithm considers the semantic background as a source of interference. Furthermore, the Distractor-aware Siamese Networks (DaSiamRPN), introduced by Zhu [23], improves the discriminatory power of the tracker by incorporating difficult negative samples in the training model. Besides, Li et al. applied the ResNet network into a SiamRPN++ visual tracker [24], which achieves good tracking results. Yet, the SiamRPN-based tracker is susceptible to drift towards regions that are similar to the target, due to the absence of model updates.

The DCF method [25,26] has demonstrated notable efficiency in training the frequency domain filter. The pioneer Minimum Output Sum of Squared Error (MOSSE) algorithm [14] adopts the gray-scale feature to complete target tracking with high execution efficiency. Yet, this algorithm does not take the background information and scale updating into consideration. Moreover, additional features, such as color name features [27], Histogram of Gradient (HOG) [15], and deep features [8], have proven to be valuable in advancing the development of tracking. To be more specific, the HOG representation is obtained by the Kernel Correlation Filter (KCF) algorithm [15]. Furthermore, a multi-feature of Visual Geometry Group (VGG) and HOG are fused with multi-cue correlation by Wang et al. [28]. However, the above DCF algorithms are sensitive to the situation of occlusion [29] during tracking. Besides, another issue that arises in visual tracking is when the target is similar in color features to its surrounding background.

From the above analysis, tracking methods based on DL and DCF have been widely applied to tracking visible targets [24,30,31]. The similarity in color between the target and background often results in tracking drift for most trackers to some extent. The spatial and spectral characteristics of hyperspectral data can effectively distinguish objects from the background, which is widely used in the field of image classification [32,33]. Specifically, Ding et al. [32] proposed a novel multi-feature fusion network, which extracts multi-scale pixel-level local features for HSI classification. Zhang et al. [33] proposed a graph neural framework based on multiple adaptive receptive fields, which addresses issues such as insufficient labeled training samples and high spectral mixing between materials. Benefiting from spectral properties, a number of effective algorithms [3,34,35,36,37] have been developed in the field of HOT. Qian et al. [4] introduced a HOT method utilizing the convolutional network without a training strategy. The DCF framework obtains a high execution efficiency, yet the elimination of redundancy in the spectral information is not considered. Uzkent et al. [38] introduced a Deep Kernelized Correlation Filter-based method (DeepHKCF) for object tracking, which did not consider the valuable spectral information. In their work, HOG feature channels were concatenated with pure hyperspectral channels for tracking aerial objects. Zhang et al. [39] proposed an HSV tracking method based on multi-feature integration(MFI-HVT), which generates feature maps by a Histogram of Gradients (HOG) and a pre-trained VGG-19 network, and then detects targets on high-speed aircraft by using a kernelized correlation filter framework. The Material-based Hyperspectral Tracking (MHT) [3] method, developed by Xiong, embeds spectral-spatial information into multi-dimensional gradient histograms. In addition, the Band Attention Aware Ensemble Network (BAENet) [34] method, introduced by Li, utilizes a band selection network combined with the VITAL [40] tracking framework. Liu et al. [41] proposed an anchor-free siamese network (called HA-Net), including RGB classification, a regression network, and a hyperspectral network. Furthermore, HA-Net performs online updates by selecting the image with the highest confidence every ten frames. However, the limited number of HSVs may result in insufficient training for the model.

In general, tracking algorithms that utilize HSVs encounter limitations related to the size of the dataset, which can make it challenging to effectively leverage the semantic information of Hyperspectral images (HSIs). In this paper, with the siamese network as the backbone model in our tracker, the overall framework also includes band selection [42], model training, feature transfer, and the template updating mechanism. Specifically, the three valuable bands are first obtained via a genetic algorithm [43] with the joint entropy or the optimal index factor. Besides, our model is pre-trained on datasets from the General Objective Tracking (GOT) benchmark [44], and the semantic information in the HSV is known to benefit from Transfer Learning (TL) [45]. In addition, a convolutional neural network named Hyperspectral template Update Network (HSUpdateNet) has been developed, inspired by the UpdateNet [46]. This method not only replaces the template manual update function, but also utilizes the spectral information for tracking. For the reason that the UpdateNet [46] is successfully applied into the Fully Convolutional Siamese Networks (SiamFC [47]) and the SiamRPN++ [24], the HSUpdateNet designed by us is applied into our previous work named BS-SiamRPN [36]. Subsequently, the material composition provided by the rich spectral information improves the ability of feature identification. Finally, the selected bands are input into the backbone network, which extracts spectral features of HSVs. The effect of the TL strategy on the proposed model can be seen in Figure 1, which proves that a tracking algorithm without the TL strategy is very susceptible to the representation of the target.

The main contributions are listed as follows.

•

The SiamRPN model is applied into the field of HOT, which verifies its applicability in processing HSVs.

•

To reduce redundancy in HSVs, bands are determined by an intelligent optimization algorithm based on maximum joint entropy.

•

TL has been effectively applied into the domain of HOT, which effectively solves the limitation of lacking labeled data in hyperspectral datasets referring to learning deep models.

•

The proposed HSUpdateNet with an effective template update strategy, which exploits rich spectral information, helps to obtain a more accurate cumulative template, and deals with the problem of deformation.

The following is the relevant content of this paper. In Section 1, the current state of development of the HSV tracking is given in detail. Then, the proposed framework is described in Section 2. In Section 3, experimental results are analyzed. In Section 4, the direction of future research is discussed. Finally, Section 5 draws several conclusions.

2. Materials and Methods

2.1. SiamRPN

The SiamRPN model converts the initial similarity calculation problem into a regression or classification problem using a large-scale offline training network, which is shown in Figure 2. The SiamRPN extracts features using a siamese network, which is in line with the architecture of the SiamFC [47] model. The region proposal network also includes a classification branch and a regression one. The former is specifically used to calculate the Intersection Over Union (IOU) of the output box with the real target. Besides, a fixed threshold is set to distinguish the target from the background. The regression branch of the proposed method is responsible for both accurately matching the predicted bounding box for the target with its actual state, as well as compensating for any potential scale variations during the initial stages of the tracking process.

The accuracy and speed of several trackers, including the CFNet tracker [48], the DaSiamRPN tracker [49], and the SiamRPN++ tracker [24], are balanced, yet several problems still need to be solved. First of all, most siamese trackers only utilize characteristics that can differentiate between foreground and non-semantic contexts. It is challenging to ensure the performance in complex environments. Secondly, siamese trackers lack an online update paradigm, which can cause the tracker to lose the target if it undergoes abrupt changes in appearance. Thirdly, the local search technique of the siamese tracker is sensitive to partially occluded targets. However, HSVs help to make up for the disadvantage of visible data in tracking.

Here, the feature of the detection frame is subjected to convolution operations by the classification branch, and the feature of the template frame refers to the regression branch.

$$A_{w\times h\times 2k}^{cls}=\varphi {\left(x\right)}_{cls}\ast \varphi {\left(z\right)}_{cls}$$

(1)

$$A_{w\times h\times 4k}^{reg}=\varphi {\left(x\right)}_{reg}\ast \varphi {\left(z\right)}_{reg}$$

(2)

where ∗ denotes the convolution operation and softmax is utilized in calculating the classification loss. $A_{w\times h\times 2k}^{cls}$ consist of $2k$ channels, indicating positive and negative excitation. Each point in the $4k$ channels of $A_{w\times h\times 4k}^{reg}$ is represented as $d_x,d_y,d_w$ and $d_h$. This is the distance between the anchor and ground truth. $\varphi {\left(x\right)}_{cls}$ and $\varphi {\left(x\right)}_{reg}$ denote the output feature in the detection branch. Both the template feature maps $\varphi {\left(z\right)}_{cls}$ and $\varphi {\left(z\right)}_{reg}$ are kernels.

The definition of the loss function $L_{reg}$ is as follows:

$$L_{reg}=\sum _{i=0}^{3}smoot{h}_{L_1}(\delta \left[i\right],\sigma )$$

(3)

$$\begin{array}{cc}\hfill \delta \left[0\right]& = (T_x-A_x)/A_w\hfill \\ \hfill \delta \left[1\right]& = (T_y-A_y)/A_h\hfill \\ \hfill \delta \left[2\right]& = ln(T_w/A_w)\hfill \\ \hfill \delta \left[3\right]& = ln(T_h/A_h)\hfill \end{array}$$

(4)

where the center point is denoted by $A_x$ and $A_y$ and the shape of anchor boxes is represented by $A_w$ and $A_h$, respectively. Moreover, the ground truth boxes are described by $T_x,T_y,T_w$, and $T_h$, respectively. $\sigma$ can control the error range of using mean square error or mean absolute error. In Equation (3), the loss ${smooth}_{L_1}$ is formulated as follows:

$${smooth}_{L_1}=\left\{\begin{array}{cc}0.5{\sigma }^2{x}^2\hfill & \mathrm{if}\left|x\right|<1/{\sigma }^2\hfill \\ \left|x\right|-1/2{\sigma }^2\hfill & \mathrm{if}\left|x\right|\ge 1/{\sigma }^2\hfill \end{array}\right.$$

(5)

Finally, the total loss function $L_{total}$ is as follows:

$$L_{total}=L_{cls}+\lambda L_{reg}$$

(6)

where $L_{cls}$ describes the cross entropy loss and $\lambda$ is a hyperparameter of the two losses ($\lambda$ = 1.2).

2.2. Hyperspectral Band Selection

In order to simplify the tracking process and reduce the redundant information of HSVs, the valuable features of HSVs are obtained by feature dimension reduction, such as feature extraction [16] and feature selection [50]. The former contains some popular methods, such as singular value decomposition and principal component analysis, which are utilized to extract features in a reduced dimensional space. However, these algorithms need to provide a new definition of feature attributes. The latter focuses on choosing features with a strong performance of classification, such as band selection on HSIs. In order to preserve the majority of the hyperspectral information, the genetic algorithm [51] is employed. As a result, the correlation value between the retained bands is usually low. Figure 3 displays the procedure of band selection.

In Figure 3, the matrix diagram of the correlation coefficient shows that the initial images can be separated into numerous groups. In addition, images belonging to the same group exhibit high correlation coefficients, and those not belonging to the same group exhibit low correlation coefficients. Therefore, it is imperative to decrease the redundancy of the original information.

With HSVs, it is time consuming to calculate the information entropy during tracking tasks. Therefore, a genetic optimization approach [52,53] is designed to swiftly identify selected bands with the highest entropy values. Given the initial image $D=[d_1;d_2;\dots ;d_{16}]\in R^{m\times n\times 16}(m=135,n=271)$, the joint information entropy H is described as

$$H(d_i,d_j,d_k)=-\sum _{I,J,K=0}^{255}{P}_{I,J,K}{log}_2{P}_{I,J,K}$$

(7)

where $P_{I,J,K}$ represents the joint probability between pixel intensity value J in band $d_j$, pixel intensity value I in band $d_i$, and pixel intensity value K in band $d_k$.

Therefore, the entropy H can be defined as a fitness function in the optimization method. Subsequently, banded arrays are used as chromosomes in the genetic algorithm, as depicted in Figure 3. Iterations of selection, crossover, and variation are carried out to determine the appropriate band. Based on correlation analysis and the genetic method, three bands (band 2, band 5, and band 8 for the initial image of the toy1 video) are obtained. Additionally, a three-spectra input is required by the fundamental deep network.

In conclusion, band selection based on joint entropy takes into account both implementation effectiveness and information parsimony. Specifically, the inclusion of additional bands may lead to redundant information within a particular spectral dataset.

2.3. Transfer Learning

Transfer Learning (TL) is widely used in machine learning areas, such as item recognition, medical diagnosis, and speech recognition [54,55,56]. This is because training DL models in specific application contexts sometimes exhibits various contradictions, such as too few training samples and the high cost of data labeling.

TL can be divided into four categories based on its processing, sample instance based, feature based, parameter sharing based, and relationship based. We use a TL based on parameter sharing, where certain parameters are shared between the model in the source domain $D_s$ of normal RGB data and the target domain $D_t$ of hyperspectral video data from the perspective of the model. The goal of migration learning is to find the prediction function F, formulated as:

$$Y=F(X,D_t,D_s)$$

(8)

where X is the test set of hyperspectral data and Y is the tracking results.

The normal RGB data are pre-trained first and then the model is fine-tuned using the hyperspectral video data. Pre-trained models that employ frequently employed RGB data source domains, encompassing vast datasets, have the potential to augment model robustness and foster superior generalization, such as ImageNet [49], COCO [57], and GOT-10K [44].

The HSIs contain spectral information, which can be lost during pre-training with RGB data. Furthermore, the TL strategy cannot be used directly with hyperspectral data, due to of its multichannel information. Thus, the band selection module is required. Finally, Figure 4 shows the flowchart of the proposed algorithm.

2.4. Hyperspectral Update Learning

For long-term tracking tasks, such as long sequences in hyperspectral datasets, hyperspectral video tracking has difficulty coping with the fact that the appearance of targets changes dramatically. The SiamRPN model updates targets’ template with the linear fusion of the current and the cumulative template, which causes information to decay exponentially over time. Its simplicity limits the potential benefits that can be obtained from learning updates, despite the fact that it can enhance tracking performance. Moreover, this linear combination method cannot utilize the potential spectral information for HOT. Therefore, a learned updating method is adopted to replace the linear mechanism. Inspired by UpdateNet [46], a convolutional neural network named HSUpdateNet is designed to obtain the optimal template for HSVs. It replaces the template manual update function and takes advantage of the spectral information. The template in SiamRPN is updated to an an exponentially decaying moving average value over time. Specifically, the recursive formula for the template update is represented as follows:

$${\tilde{T}}_p=(1-\gamma ){\tilde{T}}_{p-1}+\gamma {\tilde{T}}_p$$

(9)

where $T_p$ denotes the new template sample at frame p, and ${\tilde{T}}_{p-1}$ denotes the accumulated template. Besides, $\gamma =0.01$ is the update factor, which means that the appearance of objects changes smoothly over the subsequent images.

To exploit the spectral information of HSIs, a universal function $\psi$ is learned to overcome the limitations of simple template averaging.

The learning function HSUpdateNet $\psi$ is based on the initial true template, the last accumulated one, and the template referring to the predicted object position at frame p. $T_0^{GT}$ represents the initial frame of each HSV, which is shown in Figure 5. The HSUpdateNet is trained to obtain a prediction of the target template. The predict template ${\tilde{T}}_p$ should match the template ${\tilde{T}}_{p+1}^{GT}$, referring to the true position at frame $p+1$. To accomplish this, HSUpdateNet is trained by minimizing the Euclidean distance between real templates and updated templates at the next frame, which is defined as follows:

$$\begin{array}{cc}\hfill {\mathcal{L}}_2& ={∥\psi (T_0^{GT},{\tilde{T}}_{p-1},T_p)-T_{p+1}^{GT}∥}_2\hfill \\ & \hfill =\sqrt{{\displaystyle \sum _{p=1}^n}{(\psi (T_0^{GT},{\tilde{T}}_{p-1},T_p)-T_{p+1}^{GT})}^2}\end{array}$$

(10)

where ${\mathcal{L}}_2$ represents the Euclidean distance and n represents the length of test HSVs.

$${\tilde{T}}_p=\psi (T_0^{GT},{\tilde{T}}_{p-1},T_p)$$

(11)

Meanwhile, the multi-stage training strategy is utilized to avoid repeated training, which makes the procedure simple and efficient. It is not reasonable to use real values for ${\tilde{T}}_{p-1}$ and $T_p$ during the first training, because real templates are rarely used in real tracking. In the first phase, the original tracker with the linear updating strategy is executed on HSVs to generate the cumulative template and the actual predicted position for each frame.

$${\tilde{T}}_p^0=(1-\gamma ){\tilde{T}}_{p-1}^0+\gamma {\tilde{T}}_p^0$$

(12)

The update factor $\gamma$ corresponds to the first approximation value of HSUpdateNet. In each subsequent training phase $k\in \left\{1,\dots ,K\right\}$, the HSUpdateNet model is trained to obtain the cumulative template and object location prediction.

$${\tilde{T}}_p^k={\psi }_p^k(T_0^{GT},{\tilde{T}}_{p-1}^{k-1},T_p^{k-1})$$

(13)

The network takes the initial template $T_0^{GT}$ at each frame into account, which provides highly reliable information. Moreover, spectral information is learned, which helps to improve the tracker’s performance.

Furthermore, the linearly updated cumulative templates and HSUpdateNet are visualized in Figure 6. The cumulative template approach utilized by the HSUpdateNet model more closely reflects real-world situations when compared to the linear update strategy. However, the linearly updated cumulative template changes at an exceptionally slow rate, making it difficult to track objects with significantly altered appearances, thus presenting a significant limitation.

This is despite the fact that the UpdateNet [46] update strategy suffers from a self-learning problem, meaning that by updating with mislabeled samples, the tracker may permanently degrade and drift. However, HSUpdateNet is trained using hyperspectral datasets and more spectral information is retained during template accumulation, which enhances the tracker’s ability to identify erroneous samples.

3. Results

The experimental setup is detailed in Section 3.1, and the ablation study is given to prove the importance of several components in the proposed tracker in Section 3.2. Section 3.4 shows the quantitative comparison of deformation. In addition, qualitative and quantitative comparisons with existing trackers on the whole dataset are given in Section 3.3 and Section 3.5, respectively.

3.1. Experiment Setup

Dataset: The GOT10K dataset, which was introduced by Huang et al. [44], serves as a valuable resource for obtaining pre-trained parameters. With over 10,000 video clips and more than one million labeled bounding boxes, the dataset consists of five subcategories: animals, motor vehicles, people, assisting objects, and partially moving objects. We performed transfer learning using the HOT2022 dataset [3], which is provided by the 2022 Hyperspectral Object Tracking (HOT2022) challenge on https://www.hsitracking.com/contest/, (accessed on 13 September 2022). The HOT2022 dataset contains 40 training video sets and 35 test sets with RGB videos, HSVs, and false-color videos. The dataset authors applied weight to each band in HSVs to generate channels of color images. This method [58] generates images(false-color videos) with similar color intensity to the color images, ensuring a fair comparison with color trackers. The HSVs are acquired by a hyperspectral camera with 16 bands of wavelengths ranging from 470 nm to 620 nm, which was modeled as a snapshot VIS produced by interuniversity microelectronics center with a bandwidth of about 10 nm per band. Each video is also labeled with relevant challenge factors, which include Deformation (DEF), Scale Variation (SV), Occlusion (OCC), Illumination Variation (IV), Background Clutters (BC), Fast Motion (FM), Out-of-view (OV), Out-of-plane Rotation (OPR), In-plane Rotation (IPR), Motion Blur (MB), and Low Resolution (LR). Ablation experiments and tracking experiments are performed using all videos. Deformation experiments are performed using videos with target deformation properties.

Table 1. The details of the experimental videos.

Video	Car3	Fruit	Hand	Kangaroo	Pedestrian2	Player
Frame	331	552	184	117	363	901
Resolution	512 × 256	493 × 232	341 × 186	385 × 206	512 × 256	463 × 256
Initial Size	188 × 84	32 × 37	103 × 108	22 × 41	13 × 44	23 × 69
Challenge	LR, OCC	BC, OCC	BC, DFM	OPR, SV	DFM, OCC	IPR, SV

presents details regarding our experimental videos, while Figure 7 depicts the corresponding RGB videos. The series for Figure 7a consists of 331 frames of 512 × 256 pixels. The tracking target is a black automobile traveling from near to far on the highway. The vehicle ranges in size from 188 × 84 pixels to 13 × 9 pixels. There are concurrent SV, LR, OCC, and IV challenges in the sequence.

Table 1 has comparable details for the other five sequences. In Figure 7b, the tracked target is a green fruit with sizes ranging from 32 × 37 pixels to 26 × 25 pixels. The video shows green fruit moving randomly in front of a cluttered branch. In addition, the target tracking in Figure 7c is a palm moving back and forth in front of a yellow background and undergoing constant changes in shape and size, ranging from 103 × 108 pixels to 87 × 98 pixels. In Figure 7d, our tracking target is a kangaroo running in front of the woods, surrounded by many similar kangaroos, and its size is from 22 × 41 pixels to 19 × 33 pixels.

In Figure 7e, the object of attention is a person walking in the shadows, who is obscured by a tree during their movement, with sizes ranging from 13 × 44 pixels to 15 × 39. Finally, in Figure 7f, the tracking target is a basketball player who keeps moving and turning around. The size of the target range from 23 × 69 pixels to 34 × 101 pixels.

Evaluation Metrics: The four most popular target tracking assessment metrics [59], i.e., precision plot, success plot, average Distance Precision at a threshold of 20 Pixels (DP@20P), and Area Under the Curve (AUC), are introduced to give the performance comparison.

The success plot represents the proportion of successful frames, where the predicted tracking box overlaps the ground truth with a fixed threshold (from zero to one). Besides, the precision plot shows the proportion of frames, where the centroid difference referring to the estimated target position and the real target position is not more than a specified threshold. In addition, the curve is determined by the varying percentages that arise from different thresholds. With the initial target state, all compared trackers execute on the full test HSVs, and all results are obtained with One-Pass Evaluation (OPE).

Implementation details: The SiamRPN model [22] is pre-trained with GOT10K [44]. The learning rate is between ${10}^{-2}$ and ${10}^{-6}$. In the training model, the batch size is 128, and the epoch parameter is fixed as 50 (Stochastic Gradient Descent (SGD) for optimization). For transfer learning, the learning rate is set from ${10}^{-4}$ to ${10}^{-6}$. The first three convolutional layers are fixed for the first five training epochs and then unfreeze. A batch size of 128 and an epoch of 20 were employed during training. In the first training phase of HSUpdateNet, a linear update with an update rate of 0.0102 is used in the template generation part. Three bands of the hyperspectral dataset are considered as training samples. Furthermore, weights are initialized by the model generated in the previous stage, while the learning rate is reduced from ${10}^{-8}$ to ${10}^{-9}$ in each epoch.

3.2. Ablation Study

To verify the algorithm’s effectiveness, a baseline algorithm is first implemented by adopting only the pre-trained model without enabling the band selection module, transfer learning module, and online update module. Based on the baseline algorithm, two alternative algorithms are also implemented. First, the band selection and transfer learning modules are utilized. Secondly, based on the second method, linear template update, and HSUpdateNet template update are executed, respectively. Results on the HOT2022 dataset are shown in Figure 8. Both the second and third two algorithms perform better than the baseline. In

Table 2. Ablation study over all the test HSVs.

Method	Video Type	AUC	DP@20P
baseline	false-color	0.482	0.763
baseline-BS	HSV	0.468	0.771
baseline-BS-TL	HSV	0.533	0.845
baseline-BS-TL-L	HSV	0.535	0.820
baseline-BS-TL-H	HSV	0.575	0.861

BS denotes Band Selection, L denotes Linear, and H denotes HSUpdate.

, the AUC value of the baseline algorithm reaches 0.483. Besides, the algorithm with only the band selection module does not perform well in objective indicators. This is because hyperspectral data are different from RGB data in feature extraction, and the baseline algorithm does not understand the spectral features well. In contrast, by enabling the band selection module and transfer learning module, which effectively utilizes the informative spectral features, the tracker performance is improved by 5%. Both the strategies of linear template update and the HSUpdateNet template update improve the tracker’s performance. In general, the former is typically used by the siamese tracker, which runs fast but with limited improvement. Compare with the strategy of no template update, the performance is only improved by 0.2%. The HSUpdateNet model not only avoids the exponential decay of information arising from a linear update but also obtains spectral information that is effective for tracking, resulting in a 4.2% improvement in performance.

3.3. Qualitative Comparison

In this section, our method is compared against seven other trackers, namely MHT, MFIHVT, BAENet, DeepHKCF, BS-SiamRPN, DaSiamRPN, and SiamRPN++. Among them, DaSiamRPN and SiamRPN++ are color trackers, and the rest are the latest HS trackers. The MHT model introduces hand-crafted features, which extracts material information, to improve the tracker performance. BAENet transforms HSVs into a group of three-channel images. These altered images are run through a number of VITAL trackers to produce several weak trackers. Furthermore, an ensemble learning process is adopted to track targets. The MFIHVT algorithm generates feature maps by HOG and the pre-trained VGG19 feature, and the DCF framework is adopted to estimate the location of objects. The DeepHKCF tracker converts HSV into a three-channel image sequence, which is fed into the VGGNet for extracting features. The BS-SiamRPN algorithm lacks a template update strategy. The DaSiamRPN algorithm utilizes data augmentation to increase its performance. The SiamRPN++ algorithm completes feature extraction using a deeper network. In this paper, the false-color videos are utilized as the input of the color tracker to compare with the HS tracker.

Experimental results of all the trackers on all six experimental sequences are shown in Figure 9, Figure 10, Figure 11, Figure 12, Figure 13 and Figure 14. The image contains several various colored rectangular boxes, as shown below, and the varied colors represent different algorithms, i.e., green denotes ours, blue denotes BANet, black denotes DaSiamRPN, yellow denotes BS-SiamRPN, cyan denotes SiamRPN++, and red denotes the ground truth.

In Figure 9, the black car begins at the beginning of the road, and gradually shrinks as it travels further and further. Furthermore, when the black car is traveling down the center of the road, a white car is driving in the other direction. When the target vehicle reaches the end of the road, another vehicle is close behind it. During this time, the two vehicles gradually shrink. The BAENet algorithm is unable to accurately surround the target due to the changing aspect ratio of the black car. At frame 263, it can be found that only the BS-SiamRPN and ours get a high overlap value with true results.

As can be seen in Figure 10, the fruit moves above the leaf, resulting in size changes that are difficult to track. Because of the blockage of leaves during the subsequent target movement, all of the trackers addressed in this paper are unable to precisely obtain the target state. Therefore, BAENet and DeepHKCF can only see a portion of the target. This problem becomes more serious as targets are frequently occluded by leaves. At frame 253, it is evident that our tracker is capable of reliably locating the target, and adapting to changes in targets. It is worth mentioning that only our tracker has a decent overlap with the true result.

The palm in Figure 11 moves over a background with similar colors, causing the palm to deform due to constant flipping. During the tracking process, the target varies frequently, yet most trackers cannot react adequately to the target changes. Nevertheless, our tracker uses HSUpdate to retrieve the best-accumulated template at frame 148, making the tracker perfectly overlap with the ground truth.

Figure 12 illustrates the challenges of tracking the kangaroo, as its fast jumps cause variations in appearance and scale, making tracking difficult. Furthermore, with a similar representation feature to that of the tracking target, other kangaroos create a significant amount of background interference for object tracking. It is difficult for the DeepHKCF to completely coincide with the real target box.

Figure 13 depicts a pedestrian walking from a well-lit area to a darker region amid a background of trees. During subsequent target motion, the pedestrian becomes occluded by trees, resulting in temporary loss of the target on the screen, posing a significant challenge for the trackers. The DeepHKCF algorithm fails to obtain the target state at frame 40. At frame 266, SiamRPN++ also loses the target due to the occurrence of occlusion. At frame 360, the BS-SiamRPN algorithm and our algorithm correctly detect the target, due to the fact of that the spectral information is preserved with band selection.

Figure 14 shows an athlete dressed in white playing basketball on a court, with movements such as shooting and running leading to variations in the shape of the target. BS-SiamRPN does not perform well in the tracking task, and because of that, the tracker does not utilize the strategy of template updating. The DeepHKCF model only adopts one deep feature, which makes the target drift to the background.

3.4. Quantitative Comparison of Adversarial Deformation

Sequentially, the performance is represented in

Table 3. Performance comparison with other trackers in terms of deformation challenge.

Algorithm	Video type	AUC	DP@20P
DaSiamRPN	false-color	0.670	0.920
SiamRPN++	false-color	0.669	0.971
DeepHKCF	HSV	0.433	0.774
BS-SiamRPN	HSV	0.641	0.970
MHT	HSV	0.708	0.930
MFIHVT	HSV	0.655	0.946
BAENet	HSV	0.51	0.920
Ours	HSV	0.708	0.971

The top two values are in red and blue.

, in terms of deformation challenges. Specifically, our algorithm almost reaches the top two performances in terms of evaluation metrics. Figure 15 displays the success and precision curves of the algorithm on videos with a deformation challenge, where the area covered by the curve represents the value. The specific results are shown in Table 3.

Overall, the MHT achieved a success rate of 0.708 and a precision value of 0.930. Besides, our proposed algorithm has the same success rate, and the precision is improved by 4.1%. It is worth noting that the MHT is the state of the art. Our tracker produced better results than other algorithms in the DEF challenge, which is shown in Figure 15 and Table 3. Specifically, the precision value of the proposed algorithm is 0.708 and the success rate is 0.971, which is a significant improvement over MHT and BS-SiamRPN, referring to the DEF challenge. Compared to BS-SiamRPN, the success rate of our algorithm is 0.1% higher and the precision rate is 6.7% higher, which is because the BS-SiamRPN algorithm lacks a template update strategy. In addition, both SiamRPN++ and our algorithm achieve the highest precision, but our algorithm’s success rate is 3.9% higher. This is because our algorithm utilizes the spectral information of HSVs. Numerical results confirm that AD-SiamRPN can make good use of HSUpdateNet to obtain the best cumulative template, which helps the tracker cope with the deformation challenge.

3.5. Quantitative Comparison

Here, the quantitative result for all seven trackers is represented by the precision and the success plots. The AUC comparison of several hyperspectral trackers and visual DL trackers is shown in

Table 4. Performance comparison with other trackers.

Algorithm	Video Type	AUC	DP@20P	FPS
DaSiamRPN	false-color	0.558	0.831	48
SiamRPN++	false-color	0.529	0.834	41
DeepHKCF	HSV	0.385	0.737	2
BS-SiamRPN	HSV	0.533	0.845	55
MHT	HSV	0.584	0.876	2
MFIHVT	HSV	0.601	0.891	2
BAENet	HSV	0.616	0.876	-
Ours	HSV	0.575	0.861	35

The top one value is in red.

.

Figure 16 shows the comparative results of the trackers over the entire dataset. It can be found that our algorithm achieves high AUC scores (0.575 in the success plot and 0.861 in the precision plot). Facing difficulties such as fast motion, occlusion, same backdrop, and low resolution, the SiamRPN++ and the DaSiamRPN easily fail to detect targets. The false-color video, which loses spectral information, does not enhance the two-color trackers very well. Moreover, the complexity of HSIs necessitates the use of transfer learning techniques for effective understanding. Our tracker, which utilizes transfer learning to capture the semantic aspects of multispectral data, achieves robust performance. The MHT algorithm leverages both spectral and spatial information to effectively distinguish the target from the background by adding material features, leading to good performance. The BAENet model performs best, for the reason that it sent the band group into the VITAL tracker via the band selection network. The spectral information is retained well, which is instructive for us for future research. The DeepHKCF algorithm does not perform well, because it only extracts one type of feature. It introduces a multi-feature fusion tracking algorithm, which retains more spectral information and has good robustness to the complex background. Compared with BS-SiamRPN, a template update module, which makes full use of spectral information, helps to deal with occlusion challenges and other aspects. As a matter of fact, the overall performance is improved, the AUC score in the success plot is from 0.533 to 0.575, and the AUC score in the precision plot is from 0.845 to 0.861.

4. Discussion

While the comprehensive experiments and analyses have established the effectiveness of the proposed AD-SiamRPN, there remains a need for further discussion regarding potential avenues for future research.

•

The dearth of high-quality datasets with annotations is a major drawback of hyperspectral target tracking, potentially hindering the learning of valuable information and resulting in overfitting. To address this limitation, our future efforts will focus on enhancing the feature extraction capabilities of the tracker, perhaps by employing self-supervised approaches, such as contrast learning.

•

The band selection module eliminates the information redundancy of HSVs and retaines the physical information. It obtains high tracking speed, but the improvement in performance was not quantified. In the future, we will compare this with other feature reduction approaches to validate the effectiveness of our band selection module.

•

Experiments have demonstrated that the spectral information helps to distinguish the target from the background information, and from this, future work can proceed to design a rational network that uses the raw 16-dimensional data as input to the network to extract valid hyperspectral features.

5. Conclusions

A tracking algorithm on hyperspectral videos in this paper benefits from band selection and an improved siamese network. From the above experiments and analysis, some conclusions are given as follows.

•

Hyperspectral information helps trackers detect an object with similar color attributes from its neighborhood.

•

The high quality of the training set in the GOT database provides the basis for the generalizability of our tracking algorithm (AD-SiamPRN).

•

The band selection module retains the spectral significance of the spectral channel, and the intelligent optimization algorithm speeds up the tracking task.

•

The transfer model aids in the extraction of semantic information from HSV, hence enhancing the tracker’s performance. The template update module (HSUpdateNet) performs well with respect to issues such as deformation.

•

Future work proceeds to design a rational network that uses the raw 16-dimensional data as input to the network to extract valid hyperspectral features. Besides, the effective attention mechanism should be applied into the field of HOT.

In addition, experimental results validate that the AD-SiamPRN method performs well in HOT (with an overall success rate of 0.575 and a deformation challenge success rate of 0.708), which lays the foundation for moving target recognition.