Detection of Small Moving Objects in Long Range Infrared Videos from a Change Detection Perspective
Abstract
:1. Introduction
- We present two new target detection frameworks from a change detection perspective for small moving targets.
- The two new schemes are unsupervised approaches as compared to the deep learning approaches in the literature. This means the proposed approaches require no training data and hence are more practical.
- We demonstrated the efficacy of the proposed approaches using actual long range and low quality MWIR videos from 3500 m to 5000 m.
- We investigated the use of synthetic bands for target detection. The performance is promising as we have comparable or better detection results for 4000 m and 5000 m videos.
- We compared with two conventional approaches (frame by frame and optical flow) and yielded comparable or better performance.
2. Small Target Detection from the Change Detection Perspective
2.1. Motivation
2.2. Proposed Unsupervised Target Detection Approaches Using Change Detection
- Perform change detection using a CD algorithm between two frames.
- Apply denoising to reduce the amount of noise in the change map.
- Perform dilation to increase size and intensity of detected changes
- Use LIG to detect anomalies in each change detection map.
- Perform dilation again to make detected change more visible
2.2.1. Change Detection
Covariance Equalization (CE)
- 1.
- Compute mean and covariance of R and T as , , ,
- 2.
- Do eigen-decomposition (or SVD).
- 3.
- Do transformation.
- 4.
- The residual image between PR and PT is defined as
Chronochrome (CC)
- 1.
- Compute mean and covariance of R and T as , , ,
- 2.
- Compute cross-covariance between R and T as
- 3.
- Do transformation.
- 4.
- Compute the residual
Anomalous Change Detection (ACD)
2.2.2. Denoising
2.2.3. Dilation
2.2.4. LIG for Target Detection
2.2.5. Dilation Again after LIG
2.2.6. Generation of Synthetic Bands
EMAP
LCE
3. Experiments
3.1. Videos
3.2. Performance Metrics
3.3. Experiments to Demonstrate the Proposed Frameworks
3.3.1. Baseline Performance Using Direct Subtraction
3.3.2. Importance of LIG in the Full Standard and Alternative Workflows
3.3.3. Detection Results for 4000 m and 5000 m Videos
3.3.4. Additional Investigations Using EMAP and LCE
Results for the 3500 m Video
Results for the 4000 m Video
Results for the 5000 m Video
3.3.5. Subjective Results
3.3.6. Computational Times
3.3.7. Performance Comparison with Other Approaches
4. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Chen, C.L.P.; Li, H.; Wei, Y.; Xia, T.; Tang, Y.Y. A Local Contrast Method for Small Infrared Target Detection. IEEE Trans. Geosci. Remote Sens. 2014, 52, 574–581. [Google Scholar] [CrossRef]
- Wei, Y.; You, X.; Li, H. Multiscale patch-based contrast measure for small infrared target detection. Pattern Recognit. 2016, 58, 216–226. [Google Scholar] [CrossRef]
- Gao, C.; Meng, D.; Yang, Y.; Wang, Y.; Zhou, X.; Hauptmann, A.G. Infrared Patch-Image Model for Small Target Detection in a Single Image. IEEE Trans. Image Process. 2013, 22, 4996–5009. [Google Scholar] [CrossRef]
- Zhang, L.; Peng, L.; Zhang, T.; Cao, S.; Peng, Z. Infrared Small Target Detection via Non-Convex Rank Approximation Minimization Joint l2,1 Norm. Remote Sens. 2018, 10, 1821. [Google Scholar] [CrossRef] [Green Version]
- Zhang, H.; Zhang, L.; Yuan, D.; Chen, H. Infrared small target detection based on local intensity and gradient properties. Infrared Phys. Technol. 2018, 89, 88–96. [Google Scholar] [CrossRef]
- Chen, Y.; Zhang, G.; Ma, Y.; Kang, J.U.; Kwan, C. Small Infrared Target Detection Based on Fast Adaptive Masking and Scaling with Iterative Segmentation. IEEE Geosci. Remote Sens. Lett. 2021, 1–5. [Google Scholar] [CrossRef]
- Kwan, C.; Budavari, B. A high-performance approach to detecting small targets in long-range low-quality infrared videos. Signal Image Video Process. 2021, 1–9. [Google Scholar] [CrossRef]
- Demir, H.S.; Cetin, A.E. Co-difference based object tracking algorithm for infrared videos. In Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 25–28 September 2016; pp. 434–438. [Google Scholar]
- Kwan, C.; Gribben, D. Importance of training strategies on target detection performance using deep learning algorithms in long range infrared videos. Signal Image Process. Int. J. 2021, 12, 3. [Google Scholar]
- Kwan, C.; Gribben, D.; Budavari, B. Target Detection and Classification Performance Enhancement using Super-Resolution Infrared Videos. Signal Image Process. Int. J. 2021, 12, 33–45. [Google Scholar] [CrossRef]
- Kwan, C.; Gribben, D. Target Detection and Classification Improvements using Contrast Enhanced 16-bit Infrared Videos. Signal Image Process. Int. J. 2021, 12, 23–38. [Google Scholar] [CrossRef]
- Kong, X.; Yang, C.; Cao, S.; Li, C.; Peng, Z. Infrared Small Target Detection via Nonconvex Tensor Fibered Rank Approximation. IEEE Trans. Geosci. Remote Sens. 2021, 1–21. [Google Scholar] [CrossRef]
- Kwan, C.; Chou, B.; Yang, J.; Tran, T. Deep Learning Based Target Tracking and Classification Directly in Compressive Measurement for Low Quality Videos. Signal Image Process. Int. J. 2019, 10, 9–29. [Google Scholar] [CrossRef]
- Ma, D.; Dong, L.; Xu, W. A Method for Infrared Sea-Sky Condition Judgment and Search System: Robust Target Detection via PLS and CEDoG. IEEE Access 2021, 9, 1439–1453. [Google Scholar] [CrossRef]
- Dai, Y.; Wu, Y.; Zhou, F.; Barnard, K. Attentional Local Contrast Networks for Infrared Small Target Detection. IEEE Trans. Geosci. Remote Sens. 2021, 1–12. [Google Scholar] [CrossRef]
- Yang, P.; Dong, L.; Xu, W. Infrared Small Maritime Target Detection Based on Integrated Target Saliency Measure. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2021, 14, 2369–2386. [Google Scholar] [CrossRef]
- Pang, D.; Shan, T.; Ma, P.; Li, W.; Liu, S.; Tao, R. A Novel Spatiotemporal Saliency Method for Low-Altitude Slow Small Infrared Target Detection. IEEE Geosci. Remote Sens. Lett. 2021, 1–5. [Google Scholar] [CrossRef]
- Hou, Q.; Wang, Z.; Tan, F.; Zhao, Y.; Zheng, H.; Zhang, W. RISTDnet: Robust Infrared Small Target Detection Network. IEEE Geosci. Remote Sens. Lett. 2021, 1–5. [Google Scholar] [CrossRef]
- Du, S.; Zhang, P.; Zhang, B.; Xu, H. Weak and Occluded Vehicle Detection in Complex Infrared Environment Based on Improved YOLOv4. IEEE Access 2021, 9, 25671–25680. [Google Scholar] [CrossRef]
- Song, Z.; Yang, J.; Zhang, D.; Wang, S.; Li, Z. Semi-Supervised Dim and Small Infrared Ship Detection Network Based on Haar Wavelet. IEEE Access 2021, 9, 29686–29695. [Google Scholar] [CrossRef]
- Wan, M.; Ye, X.; Zhang, X.; Xu, Y.; Gu, G.; Chen, Q. Infrared Small Target Tracking via Gaussian Curvature-Based Compressive Convolution Feature Extraction. IEEE Geosci. Remote Sens. Lett. 2021, 1–5. [Google Scholar] [CrossRef]
- Zhao, M.; Li, W.; Li, L.; Ma, P.; Cai, Z.; Tao, R. Three-Order Tensor Creation and Tucker Decomposition for Infrared Small-Target Detection. IEEE Trans. Geosci. Remote Sens. 2021, 1–16. [Google Scholar] [CrossRef]
- Sun, H.; Liu, Q.; Wang, J.; Ren, J.; Wu, Y.; Zhao, H.; Li, H. Fusion of Infrared and Visible Images for Remote Detection of Low-Altitude Slow-Speed Small Targets. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2021, 14, 2971–2983. [Google Scholar] [CrossRef]
- Raza, A.; Liu, J.; Liu, Y.; Liu, J.; Li, Z.; Chen, X.; Huo, H.; Fang, T. IR-MSDNet: Infrared and Visible Image Fusion Based On Infrared Features and Multiscale Dense Network. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2021, 14, 3426–3437. [Google Scholar] [CrossRef]
- Xue, W.; Qi, J.; Shao, G.; Xiao, Z.; Zhang, Y.; Zhong, P. Low-Rank Approximation and Multiple Sparse Constraint Modeling for Infrared Low-Flying Fixed-Wing UAV Detection. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2021, 14, 4150–4166. [Google Scholar] [CrossRef]
- Lohit, S.; Kulkarni, K.; Turaga, P. Direct inference on compressive measurements using convolutional neural networks. In Proceedings of the 2016 IEEE International Conference on Image Processing, Phoenix, AZ, USA, 25–28 September 2016; pp. 1913–1917. [Google Scholar] [CrossRef]
- Adler, A.; Elad, M.; Zibulevsky, M. Compressed Learning: A Deep Neural Network Approach. arXiv 2016, arXiv:1610.09615v1. [Google Scholar]
- Xu, Y.; Kelly, K.F. Compressed domain image classification using a multi-rate neural network. arXiv 2019, arXiv:1901.09983. [Google Scholar]
- Wang, Z.W.; Vineet, V.; Pittaluga, F.; Sinha, S.N.; Cossairt, O.; Kang, S.B. Privacy-Preserving Action Recognition Using Coded Aperture Videos. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA, 16–17 June 2019; pp. 1–10. [Google Scholar]
- Vargas, H.; Fonseca, Y.; Arguello, H. Object Detection on Compressive Measurements using Correlation Filters and Sparse Representation. In Proceedings of the 2018 26th European Signal Processing Conference (EUSIPCO), Rome, Italy, 3–7 September 2018; pp. 1960–1964. [Google Scholar]
- Degerli, A.; Aslan, S.; Yamac, M.; Sankur, B.; Gabbouj, M. Compressively Sensed Image Recognition. In Proceedings of the 2018 7th European Workshop on Visual Information Processing (EUVIP), Tampere, Finland, 26–28 November 2018; pp. 1–6. [Google Scholar]
- Latorre-Carmona, P.; Traver, V.J.; Sánchez, J.S.; Tajahuerce, E. Online reconstruction-free single-pixel image classification. Image Vis. Comput. 2019, 86, 28–37. [Google Scholar] [CrossRef]
- Kwan, C.; Gribben, D.; Chou, B.; Budavari, B.; Larkin, J.; Rangamani, A.; Tran, T.; Zhang, J.; Etienne-Cummings, R. Real-Time and Deep Learning Based Vehicle Detection and Classification Using Pixel-Wise Code Exposure Measurements. Electronics 2020, 9, 1014. [Google Scholar] [CrossRef]
- Kwan, C.; Budavari, B. Enhancing Small Moving Target Detection Performance in Low-Quality and Long-Range Infrared Videos Using Optical Flow Techniques. Remote Sens. 2020, 12, 4024. [Google Scholar] [CrossRef]
- Schaum, A.P.; Stocker, A. Hyperspectral change detection and supervised matched filtering based on covariance equalization. Def. Secur. 2004, 5425, 77–90. [Google Scholar] [CrossRef]
- Schaum, A.; Stocker, A. Long-interval chronochrome target detection. Int. Symp. Spectral Sens. Res. 1997, 1760–1770. [Google Scholar]
- Theiler, J.; Perkins, S. Proposed framework for anomalous change detection. In ICML Workshop on Machine Learning Algorithms for Surveillance and Event Detection; Association for Computing Machinery: New York, NY, USA, 2006. [Google Scholar]
- Bernabe, S.; Marpu, P.R.; Plaza, A.; Mura, M.D.; Benediktsson, J.A. Spectral–Spatial Classification of Multispectral Images Using Kernel Feature Space Representation. IEEE Geosci. Remote Sens. Lett. 2013, 11, 288–292. [Google Scholar] [CrossRef]
- Bernabé, S.; Marpu, P.R.; Plaza, A.; Benediktsson, J.A. Spectral unmixing of multispectral satellite images with dimensionality expansion using morphological profiles. In Proceedings of the Satellite Data Compression, Communications, and Processing VIII, San Diego, CA, USA, 19 October 2012; Volume 8514, p. 85140Z. [Google Scholar]
- Mura, M.D.; Benediktsson, J.A.; Waske, B.; Bruzzone, L. Morphological Attribute Profiles for the Analysis of Very High Resolution Images. IEEE Trans. Geosci. Remote Sens. 2010, 48, 3747–3762. [Google Scholar] [CrossRef]
- Mura, M.D.; Benediktsson, J.A.; Waske, B.; Bruzzone, L. Extended profiles with morphological attribute filters for the analysis of hyperspectral data. Int. J. Remote Sens. 2010, 31, 5975–5991. [Google Scholar] [CrossRef]
- Dao, M.; Kwan, C.; Bernabe, S.; Plaza, A.J.; Koperski, K. A Joint Sparsity Approach to Soil Detection Using Expanded Bands of WV-2 Images. IEEE Geosci. Remote Sens. Lett. 2019, 16, 1869–1873. [Google Scholar] [CrossRef]
- Xia, C.; Li, X.; Zhao, L. Infrared Small Target Detection via Modified Random Walks. Remote Sens. 2018, 10, 2004. [Google Scholar] [CrossRef] [Green Version]
- Ayhan, B.; Kwan, C. A New Approach to Change Detection Using Heterogeneous Images. In Proceedings of the 2019 IEEE 10th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), New York, NY, USA, 10–12 October 2019. [Google Scholar]
- Kwan, C.; Zhou, J. High performance change detection in hyperspectral images using multiple references. In Proceedings of the Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XXIV, Orlando, FL, USA, 15–19 April 2018; Volume 10644, p. 106440Z. [Google Scholar]
- DSIAC Dataset. Available online: https://blogs.upm.es/gti-work/2013/05/06/sensiac-dataset-for-automatic-target-recognition-in-infrared-imagery/ (accessed on 16 August 2021).
- Mathworks. 2020. Available online: https://www.mathworks.com/help/images/ref/imdiffusefilt.html (accessed on 6 July 2021).
- MOT Challenge. Available online: https://motchallenge.net/ (accessed on 6 July 2021).
- Li, K.; Wan, G.; Cheng, G.; Meng, L.; Han, J. Object detection in optical remote sensing images: A survey and a new benchmark. ISPRS J. Photogramm. Remote Sens. 2020, 159, 296–307. [Google Scholar] [CrossRef]
- Li, K.; Cheng, G.; Bu, S.; You, X. Rotation-Insensitive and Context-Augmented Object Detection in Remote Sensing Images. IEEE Trans. Geosci. Remote Sens. 2018, 56, 2337–2348. [Google Scholar] [CrossRef]
- Cheng, G.; Zhou, P.; Han, J. Learning Rotation-Invariant Convolutional Neural Networks for Object Detection in VHR Optical Remote Sensing Images. IEEE Trans. Geosci. Remote Sens. 2016, 54, 7405–7415. [Google Scholar] [CrossRef]
TP | FP | MD | P | R | F1 |
---|---|---|---|---|---|
312 | 1494 | 0 | 0.1727 | 1 | 0.2945 |
TP | FP | MD | P | R | F1 |
---|---|---|---|---|---|
293 | 5 | 7 | 0.9832 | 0.9767 | 0.9799 |
TP | FP | MD | P | R | F1 |
---|---|---|---|---|---|
296 | 834 | 4 | 0.2619 | 0.9867 | 0.4139 |
CD Method | TP | FP | MD | P | R | F1 |
---|---|---|---|---|---|---|
ACD | 298 | 120 | 2 | 0.7129 | 0.9933 | 0.83 |
CC | 303 | 500 | 0 | 0.3773 | 1 | 0.5437 |
CE | 301 | 497 | 0 | 0.3772 | 1 | 0.5478 |
(a) Standard Approach | ||||||
---|---|---|---|---|---|---|
CD Method | TP | FP | MD | P | R | F1 |
ACD | 297 | 1 | 3 | 0.9966 | 0.99 | 0.9933 |
CC | 297 | 1 | 3 | 0.9966 | 0.99 | 0.9933 |
CE | 297 | 1 | 3 | 0.9966 | 0.99 | 0.9933 |
(b) Alternate Approach | ||||||
CD Method | TP | FP | MD | P | R | F1 |
ACD | 189 | 135 | 111 | 0.5833 | 0.63 | 0.6058 |
CC | 309 | 16 | 0 | 0.9508 | 1 | 0.9747 |
CE | 313 | 238 | 0 | 0.5680 | 1 | 0.7246 |
(a) Standard Approach | ||||||
---|---|---|---|---|---|---|
CD Method | TP | FP | MD | P | R | F1 |
ACD | 295 | 3 | 5 | 0.9899 | 0.9833 | 0.9866 |
CC | 297 | 1 | 3 | 0.9966 | 0.99 | 0.9933 |
CE | 297 | 1 | 3 | 0.9966 | 0.99 | 0.9933 |
(b) Alternate Approach | ||||||
CD Method | TP | FP | MD | P | R | F1 |
ACD | 270 | 84 | 30 | 0.7627 | 0.90 | 0.8257 |
CC | 362 | 4 | 0 | 0.9891 | 1 | 0.9945 |
CE | 423 | 10 | 0 | 0.9769 | 1 | 0.9883 |
(a) Standard Approach | ||||||
---|---|---|---|---|---|---|
CD Method | TP | FP | MD | P | R | F1 |
ACD | 288 | 10 | 12 | 0.9664 | 0.96 | 0.9484 |
CC | 288 | 10 | 12 | 0.9664 | 0.96 | 0.9484 |
CE | 288 | 10 | 12 | 0.9664 | 0.96 | 0.9484 |
(b) Alternate Approach | ||||||
CD Method | TD | FP | MD | P | R | F1 |
ACD | 281 | 65 | 19 | 0.8121 | 0.9367 | 0.8699 |
CC | 330 | 19 | 0 | 0.9456 | 1 | 0.9720 |
CE | 337 | 30 | 0 | 0.9183 | 1 | 0.9574 |
Image Used | CD Method | TP | FP | MD | P | R | F1 |
---|---|---|---|---|---|---|---|
Original | ACD | 297 | 1 | 3 | 0.9966 | 0.990 | 0.9933 |
Original | CC | 297 | 1 | 3 | 0.9966 | 0.990 | 0.9933 |
Original | CE | 297 | 1 | 3 | 0.9966 | 0.990 | 0.9933 |
EMAP | ACD | 288 | 10 | 12 | 0.9664 | 0.960 | 0.9632 |
EMAP | CC | 267 | 31 | 23 | 0.8959 | 0.890 | 0.8929 |
EMAP | CE | 291 | 7 | 9 | 0.9765 | 0.970 | 0.9732 |
LCE5 | ACD | 293 | 5 | 7 | 0.9832 | 0.9767 | 0.9799 |
LCE5 | CC | 291 | 7 | 9 | 0.9765 | 0.970 | 0.9732 |
LCE5 | CE | 294 | 4 | 6 | 0.9866 | 0.980 | 0.9833 |
Image Used | CD Method | TP | FP | MD | P | R | F1 |
---|---|---|---|---|---|---|---|
Original | ACD | 295 | 3 | 5 | 0.9899 | 0.9833 | 0.9866 |
Original | CC | 297 | 1 | 3 | 0.9966 | 0.990 | 0.9933 |
Original | CE | 297 | 1 | 3 | 0.9966 | 0.990 | 0.9933 |
EMAP | ACD | 297 | 1 | 3 | 0.9966 | 0.990 | 0.9933 |
EMAP | CC | 284 | 16 | 16 | 0.9467 | 0.9467 | 0.9467 |
EMAP | CE | 297 | 1 | 3 | 0.9966 | 0.990 | 0.9933 |
LCE5 | ACD | 295 | 3 | 5 | 0.9899 | 0.9833 | 0.9866 |
LCE5 | CC | 295 | 3 | 5 | 0.9899 | 0.9833 | 0.9866 |
LCE5 | CE | 296 | 2 | 4 | 0.9933 | 0.9867 | 0.985 |
Image Used | CD Method | TP | FP | MD | P | R | F1 |
---|---|---|---|---|---|---|---|
Original | ACD | 288 | 10 | 12 | 0.9664 | 0.96 | 0.9484 |
Original | CC | 288 | 10 | 12 | 0.9664 | 0.96 | 0.9484 |
Original | CE | 288 | 10 | 12 | 0.9664 | 0.96 | 0.9484 |
EMAP | ACD | 283 | 16 | 17 | 0.9465 | 0.9433 | 0.9449 |
EMAP | CC | 268 | 33 | 32 | 0.8904 | 0.8933 | 0.9071 |
EMAP | CE | 277 | 22 | 23 | 0.9264 | 0.9233 | 0.9249 |
LCE5 | ACD | 288 | 11 | 12 | 0.9832 | 0.96 | 0.9715 |
LCE5 | CC | 288 | 10 | 12 | 0.9664 | 0.96 | 0.9484 |
LCE5 | CE | 289 | 9 | 11 | 0.9698 | 0.9633 | 0.9666 |
Change Detection (s) | LIG (s) | Denoising (s) | Dilation (s) | |
---|---|---|---|---|
Standard | 70 | <1 | <1 | <1 |
Alternative | 140 | <1 | <1 | <1 |
3500 m | 4000 m | 5000 m | |||||||
---|---|---|---|---|---|---|---|---|---|
P | R | F1 | P | R | F1 | P | R | F1 | |
Single frame [7] | 0.941 | 0.95 | 0.945 | 0.943 | 0.943 | 0.943 | 0.928 | 0.943 | 0.936 |
Optical flow [34] | 0.969 | 0.969 | 0.969 | 0.928 | 0.928 | 0.928 | 0.986 | 0.986 | 0.986 |
Proposed (standard with CC) | 0.9966 | 0.990 | 0.9933 | 0.9966 | 0.990 | 0.9933 | 0.9664 | 0.96 | 0.9484 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Kwan, C.; Larkin, J. Detection of Small Moving Objects in Long Range Infrared Videos from a Change Detection Perspective. Photonics 2021, 8, 394. https://doi.org/10.3390/photonics8090394
Kwan C, Larkin J. Detection of Small Moving Objects in Long Range Infrared Videos from a Change Detection Perspective. Photonics. 2021; 8(9):394. https://doi.org/10.3390/photonics8090394
Chicago/Turabian StyleKwan, Chiman, and Jude Larkin. 2021. "Detection of Small Moving Objects in Long Range Infrared Videos from a Change Detection Perspective" Photonics 8, no. 9: 394. https://doi.org/10.3390/photonics8090394