MATI: Multimodal Adaptive Tracking Integrator for Robust Visual Object Tracking
Abstract
:1. Introduction
2. Related Works
2.1. Single-Modal Tracking
2.2. Multimodal Tracking
3. Materials and Methods
3.1. Network Architecture
3.1.1. Adaptive Mechanisms for Multimodal Tracking
3.1.2. Crossmodal Compensation Module
3.1.3. Token Spatial Filtering Module
3.2. Theoretical Model and Evaluation Metrics for Simulating Device Behavior
3.2.1. Device Characteristics
3.2.2. Relationship Modeling and Construction
3.2.3. Tracking Evaluation
Notation
- : the ground truth bounding box co-ordinates, where and denote the top-left corner co-ordinates, and and represent the width and height, respectively.
- : the predicted bounding box co-ordinates, with and indicating the top-left corner.
- : the center co-ordinates of the bounding box, which are calculated as the midpoint between the bounding box corners.
Displacement Vectors
Threshold Parameters
- the maximum allowable displacement error threshold over a sequence of frames;
- the accumulative error threshold over a time window;
- the threshold that determines the acceptability of the displacement error.
Tracking Metrics
- The first condition implies successful tracking if no single frame exceeds the displacement error threshold;
- The second condition ensures that, if any frame meets the single-frame evaluation criterion, then subsequent frames must satisfy both robustness and accumulative evaluations.
3.3. Multimodal Aircraft Tracking Dataset
3.3.1. Background
3.3.2. Dataset Description
3.3.3. Dataset Configuration
- low-light processing: Given that the dataset already includes images from different lighting conditions, to further enhance the model’s robustness under extreme low-light conditions, we intensively processed a subset of images. In each phase of flight—ground movement, take-off ascent, and high-altitude—25% of the image frames were selected randomly for low-light preprocessing. This strategy ensures an even distribution of low-light images throughout the dataset, helping to prevent model overfitting to specific lighting conditions and enhancing the model’s generalization to varying light environments. Low-light preprocessing techniques mainly include brightness adjustment to simulate images captured in inadequately lit environments and contrast enhancement to better distinguish targets from backgrounds amid low visibility.
- Data registration: In constructing the multimodal dataset, we emphasized the importance of imperfect data alignment. Our dataset is manually processed to ensure that targets in visible and infrared images are closely aligned, which is crucial for effective tracking. Although the shapes and sizes may not be perfectly consistent due to differences in camera optics and focal lengths, this type of imperfect registration is more representative of real-world multisensor imaging scenarios. This registration approach reflects our deep understanding of real-world multimodal data characteristics. In practice, differences in sensor installation, angles, and focal distances often result in variations in the images’ shape and size. By accurately aligning only the target positions, our model better adapts to this diversity, learning how to effectively track targets under varying sensor characteristics.
- Data annotation: Each frame of the visible and infrared sequences was meticulously annotated through two rounds of manual review, ensuring precise dataset annotations. We utilized the COCO [34] dataset’s annotation format, which includes the co-ordinates of each target’s bounding box (x, y, w, h). This standardization facilitates cross-dataset comparisons and aids in model transfer learning. Additionally, aligning with the GOT-10k [35] format, we incorporated several crucial annotation files to document target status and attributes. Absence.label indicates target visibility per frame, marking ’1’ for absence and ’0’ for presence. Cover.label quantifies the degree of occlusion on a scale from 0 (fully occluded) to 1 (fully visible), with an unobstructed view marked as 8. Cut_by_image.label identifies if a target is truncated by the image border.
4. Experiments and Results
4.1. Experiment Setup
4.1.1. Experimental Environment and Datasets
4.1.2. Model Configuration and Training Details
4.1.3. Input Data Handling
4.2. Comparative Study and Analysis of TSF Module
4.2.1. Experimental Setup and Datasets
4.2.2. Performance Evaluation
- SR0.5, SR0.75, and SR0.9: these metrics represent the success rates at thresholds of 0.5, 0.75, and 0.9, respectively, indicating the proportion of tracking instances where the intersection over union (IoU) between the predicted and ground truth bounding boxes exceeds these thresholds.
- Average overlap ratio (AOR): this measures the average IoU between the tracking and ground truth bounding boxes.
- Average precision error (APE): this reflects the average error in the location of the predicted bounding boxes.
4.2.3. Visualization Analysis
Further Visualizations
4.3. Ablation Study and Analysis of CMC Module
4.3.1. Module Design and Integration in Vision Transformer (ViT) Backbone
4.3.2. Computational Efficiency Evaluation
4.3.3. Performance Evaluation on Public Datasets
Training Strategies
- Strategy A: only the top-layer CMCe and the intermediate TSF modules were unfrozen, with the CMC module added to the middle blocks.
- Strategy B: only the top-layer CMCe was unfrozen, with the CMC module added to the middle blocks.
- Strategy C: only the top-layer CMCe was unfrozen, with no additional module added.
- Strategy D: the top-layer CMC and intermediate TSF modules were unfrozen, with the CMC module added to the middle blocks.
- Strategy E: only the top-layer CMC was unfrozen, with the CMC module added to the middle blocks.
- Strategy F: only the top-layer CMC was unfrozen, with no additional module added.
Performance Metrics and Results
- Mean precision rate (MPR): this quantifies the accuracy with which a tracker can localize the target in terms of the predicted bounding box precision against the ground truth.
- Mean success rate (MSR): this measures the proportion of successful tracking instances whereby the predicted bounding box overlaps with the ground truth by a threshold commonly accepted in tracking benchmarks.
4.3.4. Performance Evaluation on Private Datasets
- Tracking success (TS): this metric evaluates whether the tracking was successful across the entire image sequence. Refer to Equation (16) for detailed computation methods.
- Number of false detections (NFDs): counts the instances when the model erroneously identifies nontargets as targets during the tracking sequence. The criteria for determining a false detection are detailed in Equation (13).
- Relative co-ordinate offset proportion (RCOP): this metric evaluates the maximum value from a list of normalized displacement errors across all frames in a tracking sequence. Each frame’s displacement error, , is normalized by the area of the target bounding box, , as shown in Equation (13). RCOP is defined by the highest normalized ratio obtained during the sequence. Lower RCOP values are beneficial, indicating minimal deviation from the target’s ideal trajectory, thus reducing the need for manual adjustments and ensuring greater operational stability of the electro-optical theodolite systems.
5. Discussion and Conclusions
5.1. Overview of Key Findings
5.2. In-Depth Discussion of Modules
5.2.1. TSF Module Performance Variations
5.2.2. Challenges with the CMCe Module
5.2.3. Placement of CMC Modules
5.3. Future Directions and Potential Enhancements
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Appendix A
References
- Wang, D.; Wang, J.G.; Xu, K. Deep learning for object detection, classification and tracking in industry applications. Sensors 2021, 21, 7349. [Google Scholar] [CrossRef] [PubMed]
- Danelljan, M.; Hager, G.; Shahbaz Khan, F.; Felsberg, M. Learning Spatially Regularized Correlation Filters for Visual Tracking. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 4310–4318. [Google Scholar]
- Kiani Galoogahi, H.; Fagg, A.; Lucey, S. Learning Background-Aware Correlation Filters for Visual Tracking. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 1135–1143. [Google Scholar]
- Bertinetto, L.; Valmadre, J.; Henriques, J.F.; Vedaldi, A.; Torr, P.H. Fully-Convolutional Siamese Networks for Object Tracking. In Proceedings of the Computer Vision–ECCV 2016 Workshops, Amsterdam, The Netherlands, 8–10 and 15–16 October 2016; Proceedings, Part II 14. Springer: Berlin/Heidelberg, Germany, 2016; pp. 850–865. [Google Scholar]
- Li, B.; Yan, J.; Wu, W.; Zhu, Z.; Hu, X. High Performance Visual Tracking with Siamese Region Proposal Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 8971–8980. [Google Scholar]
- Han, Y.; Deng, C.; Zhang, Z.; Nan, J.; Zhao, B. State-aware anti-drift robust correlation tracking. arXiv 2018, arXiv:1806.10759. [Google Scholar]
- Nam, H.; Han, B. Learning Multi-Domain Convolutional Neural Networks for Visual Tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 4293–4302. [Google Scholar]
- Danelljan, M.; Bhat, G.; Khan, F.S.; Felsberg, M. Atom: Accurate Tracking by Overlap Maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 5–20 June 2019; pp. 4660–4669. [Google Scholar]
- Yang, T.; Chan, A.B. Learning Dynamic Memory Networks for Object Tracking. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 152–167. [Google Scholar]
- Chen, X.; Yan, B.; Zhu, J.; Wang, D.; Yang, X.; Lu, H. Transformer Tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Virtual, 19–25 June 2021; pp. 8126–8135. [Google Scholar]
- Wang, N.; Zhou, W.; Wang, J.; Li, H. Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Virtual, 19–25 June 2021; pp. 1571–1580. [Google Scholar]
- Yan, B.; Peng, H.; Fu, J.; Wang, D.; Lu, H. Learning Spatio-Temporal Transformer for Visual Tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Virtual, 11–17 October 2021; pp. 10448–10457. [Google Scholar]
- Lin, L.; Fan, H.; Zhang, Z.; Xu, Y.; Ling, H. Swintrack: A simple and strong baseline for transformer tracking. Adv. Neural Inf. Process. Syst. 2022, 35, 16743–16754. [Google Scholar]
- Ye, B.; Chang, H.; Ma, B.; Shan, S.; Chen, X. Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework. In Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel, 23–27 October 2022; Springer: Cham, Switzerland, 2022; pp. 341–357. [Google Scholar]
- Zhu, Y.; Li, C.; Luo, B.; Tang, J.; Wang, X. Dense Feature Aggregation and Pruning for RGBT Tracking. In Proceedings of the 27th ACM International Conference on Multimedia, Nice, France, 21–25 October 2019; pp. 465–472. [Google Scholar]
- Zhang, D.; Danelljan, M.; Gonzalez-Garcia, A.; Van De Weijer, J.; Shahbaz Khan, F. Multi-Modal Fusion for End-to-End RGB-T Tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, Seoul, Republic of Korea, 27–28 October 2019. [Google Scholar]
- Zhang, P.; Zhao, J.; Bo, C.; Wang, D.; Lu, H.; Yang, X. Jointly modeling motion and appearance cues for robust RGB-T tracking. IEEE Trans. Image Process. 2021, 30, 3335–3347. [Google Scholar] [CrossRef] [PubMed]
- Zhang, T.; Liu, X.; Zhang, Q.; Han, J. SiamCDA: Complementarity-and distractor-aware RGB-T tracking based on Siamese network. IEEE Trans. Circuits Syst. Video Technol. 2021, 32, 1403–1417. [Google Scholar] [CrossRef]
- Guo, C.; Xiao, L. High Speed and Robust RGB-Thermal Tracking via Dual Attentive Stream Siamese Network. In Proceedings of the IGARSS 2022–2022 IEEE International Geoscience and Remote Sensing Symposium, Kuala Lumpur, Malaysia, 17–22 July 2022; pp. 803–806. [Google Scholar]
- Zhu, Y.; Li, C.; Luo, B.; Tang, J. FANet: Quality-aware feature aggregation network for robust RGB-T tracking. arXiv 2018, arXiv:1811.09855. [Google Scholar]
- Long Li, C.; Lu, A.; Hua Zheng, A.; Tu, Z.; Tang, J. Multi-Adapter RGBT Tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, Seoul, Republic of Korea, 27–28 October 2019. [Google Scholar]
- Li, C.; Zhu, C.; Huang, Y.; Tang, J.; Wang, L. Cross-Modal Ranking with Soft Consistency and Noisy Labels for Robust RGB-T Tracking. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 808–823. [Google Scholar]
- Lan, X.; Ye, M.; Zhang, S.; Yuen, P. Robust Collaborative Discriminative Learning for RGB-Infrared Tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, 2–7 February 2018; Volume 32. [Google Scholar]
- Zheng, L.; Wang, S.; Tian, L.; He, F.; Liu, Z.; Tian, Q. Query-Adaptive Late Fusion for Image Search and Person Re-Identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 1741–1750. [Google Scholar]
- Zhang, P.; Zhao, J.; Wang, D.; Lu, H.; Ruan, X. Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 19–20 June 2022; pp. 8886–8895. [Google Scholar]
- Cao, B.; Guo, J.; Zhu, P.; Hu, Q. Bi-Directional Adapter for Multimodal Tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada, 20–27 February 2024; Volume 38, pp. 927–935. [Google Scholar]
- Luo, Y.; Guo, X.; Feng, H.; Ao, L. RGB-T Tracking via Multi-Modal Mutual Prompt Learning. arXiv 2023, arXiv:2308.16386. [Google Scholar]
- Hui, T.; Xun, Z.; Peng, F.; Huang, J.; Wei, X.; Wei, X.; Dai, J.; Han, J.; Liu, S. Bridging Search Region Interaction with Template for rgb-t Tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 17–24 June 2023; pp. 13630–13639. [Google Scholar]
- Sun, D.; Pan, Y.; Lu, A.; Li, C.; Luo, B. Transformer RGBT Tracking with Spatio-Temporal Multimodal Tokens. arXiv 2024, arXiv:2401.01674. [Google Scholar] [CrossRef]
- Ding, Z.; Li, H.; Hou, R.; Liu, Y.; Xie, S.; Zhou, D.; Cao, J. X Modality Assisting RGBT Object Tracking. arXiv 2023, arXiv:2312.17273. [Google Scholar]
- Zhang, X.; Demiris, Y. Self-Supervised RGB-T Tracking with Cross-Input Consistency. arXiv 2023, arXiv:2301.11274. [Google Scholar]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is All You Need. In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 4–9 December 2017; Volume 30. [Google Scholar]
- Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An image is worth 16 × 16 words: Transformers for image recognition at scale. arXiv 2020, arXiv:2010.11929. [Google Scholar]
- Lin, T.Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; Zitnick, C.L. Microsoft Coco: Common Objects in Context. In Proceedings of the Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, 6–12 September 2014; Proceedings, Part V 13. Springer: Berlin/Heidelberg, Germany, 2014; pp. 740–755. [Google Scholar]
- Huang, L.; Zhao, X.; Huang, K. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE Trans. Pattern Anal. Mach. Intell. 2019, 43, 1562–1577. [Google Scholar] [CrossRef]
- Wu, Y.; Lim, J.; Yang, M.H. Online Object Tracking: A Benchmark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013; pp. 2411–2418. [Google Scholar]
- Li, C.; Cheng, H.; Hu, S.; Liu, X.; Tang, J.; Lin, L. Learning collaborative sparse representation for grayscale-thermal tracking. IEEE Trans. Image Process. 2016, 25, 5743–5756. [Google Scholar] [CrossRef]
- Li, C.; Liang, X.; Lu, Y.; Zhao, N.; Tang, J. RGB-T object tracking: Benchmark and baseline. Pattern Recognit. 2019, 96, 106977. [Google Scholar] [CrossRef]
- He, K.; Chen, X.; Xie, S.; Li, Y.; Dollár, P.; Girshick, R. Masked Autoencoders Are Scalable Vision Learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; pp. 16000–16009. [Google Scholar]
- Danelljan, M.; Robinson, A.; Shahbaz Khan, F.; Felsberg, M. Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking. In Proceedings of the Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, 11–14 October 2016; Proceedings, Part V 14. Springer: Berlin/Heidelberg, Germany, 2016; pp. 472–488. [Google Scholar]
- Danelljan, M.; Bhat, G.; Shahbaz Khan, F.; Felsberg, M. Eco: Efficient Convolution Operators for Tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 6638–6646. [Google Scholar]
- Kim, H.U.; Lee, D.Y.; Sim, J.Y.; Kim, C.S. Sowp: Spatially Ordered and Weighted Patch Descriptor for Visual Tracking. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 3011–3019. [Google Scholar]
- Lukezic, A.; Vojir, T.; Čehovin Zajc, L.; Matas, J.; Kristan, M. Discriminative Correlation Filter with Channel and Spatial Reliability. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 6309–6318. [Google Scholar]
- Henriques, J.F.; Caseiro, R.; Martins, P.; Batista, J. High-speed tracking with kernelized correlation filters. IEEE Trans. Pattern Anal. Mach. Intell. 2014, 37, 583–596. [Google Scholar] [CrossRef] [PubMed]
Dataset | SR0.5 | SR0.75 | SR0.9 | AOR | APE |
---|---|---|---|---|---|
Pre/Post | Pre/Post | Pre/Post | Pre/Post | Pre/Post | |
GOT-10k | 0.95/0.95 | 0.86/0.85 (−1%↓) | 0.58/0.53 (−8.6%↓) | 0.85/0.84 (−1%↓) | 26.09/26.90 (+3%↓) |
OTB-100 | 0.93/0.95 (+2%↑) | 0.63/0.68 (+8%↑) | 0.13/0.15 (+15%↑) | 0.76/0.78 (+2.6%↑) | 8.57/5.88 (−31%↑) |
MATD | 0.98/0.99 (+1%↑) | 0.74/0.85 (+15%↑) | 0.38/0.45 (+18%↑) | 0.83/0.86 (+3.6%↑) | 9.57/6.71 (−30%↑) |
Configuration | MACs (G) | Change | PARAMs (M) | Change | FPS |
---|---|---|---|---|---|
Base (single-mode) | 21.52 | - | 92.12 | - | 221.97 |
CMCe+CMC (top and middle) | 23.22 | +7.9% | 118.11 | +28% | 197.74 |
CMC+CMC (top and middle) | 23.07 | +7.2% | 115.74 | +26% | 200.57 |
CMCe (top) | 22.08 | +2.6% | 100.39 | +8.9% | 208.22 |
CMC (top) | 21.93 | +1.9% | 98.03 | +6.4% | 216.95 |
Configuration | MPR | MSR |
---|---|---|
CMCe+TSF+CMC | 0.69 | 0.57 |
CMC+TSF+CMC | 0.73 | 0.63 |
CMCe+CMC | 0.72 | 0.64 |
CMC+CMC | 0.75 | 0.68 |
CMCe only | 0.76 | 0.68 |
CMC only | 0.78 | 0.71 |
SiamCSR [19] | MDNet [7] | C-COT [40] | ECO [41] | SOWP [42] | SRDCF [2] | CSR-DCF [43] | KCF [44] | Ours | |
---|---|---|---|---|---|---|---|---|---|
MPR | 0.76 | 0.72 | 0.71 | 0.70 | 0.70 | 0.64 | 0.62 | 0.46 | 0.78 |
MSR | 0.67 | 0.62 | 0.62 | 0.61 | 0.60 | 0.56 | 0.55 | 0.40 | 0.71 |
Configuration | TS | NFDs | RCOP (%) | MPR | MSR |
---|---|---|---|---|---|
Single-mode | False | - | 3.12 | 0.69 | 0.44 |
CMCe+CMC | True | 0 | 1.7 | 0.77 | 0.62 |
CMC+CMC | True | 0 | 1.12 | 0.92 | 0.85 |
CMCe only | True | 0 | 1.51 | 0.75 | 0.57 |
CMC only | True | 1 | 1.41 | 0.91 | 0.87 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Li, K.; Cai, L.; He, G.; Gong, X. MATI: Multimodal Adaptive Tracking Integrator for Robust Visual Object Tracking. Sensors 2024, 24, 4911. https://doi.org/10.3390/s24154911
Li K, Cai L, He G, Gong X. MATI: Multimodal Adaptive Tracking Integrator for Robust Visual Object Tracking. Sensors. 2024; 24(15):4911. https://doi.org/10.3390/s24154911
Chicago/Turabian StyleLi, Kai, Lihua Cai, Guangjian He, and Xun Gong. 2024. "MATI: Multimodal Adaptive Tracking Integrator for Robust Visual Object Tracking" Sensors 24, no. 15: 4911. https://doi.org/10.3390/s24154911