A Hybrid Approach to Improve the Video Anomaly Detection Performance of Pixel- and Frame-Based Techniques Using Machine Learning Algorithms
Abstract
:1. Introduction
2. Literature Review
2.1. Using Methods and Algorithm Types
2.2. Using Data set Types
2.3. Sample Video Anomaly Detection Techniques and Application Types
3. Methodology
3.1. Modeling and Algorithm Selection
3.2. Model Architecture Diagram
- (a)
- Input layer;
- (b)
- Video surveillance and anomaly detection process layer;
- (c)
- Output layer.
3.2.1. Input Layer
3.2.2. Video Surveillance and Anomaly Detection Process Layer
- -
- Anomaly detection with frame-based (kNN, SVM) algorithms;
- -
- Anomaly detection with pixel-based (motion influence map; MIM) algorithms.
3.2.3. Output Layer
3.3. Frame-Based Video Anomaly Detection (FBVAD) Method
3.3.1. Frame-Based Feature Extraction
- (i)
- Obtaining Frames
- (ii)
- Creating Key Frames/Vectors
- (iii)
- Matching Vectors
- (iv)
- Detecting Features
- (v)
- Feature Matching Matrix
- (vi)
- Optical Flow
- (vii)
- Feature Description
3.3.2. Video Anomaly Detection with k-Nearest Neighbors (kNN) Algorithm
Algorithm 1 k-Nearest Neighbors (kNN) |
1. 2. 3. 4. 5. |
3.3.3. Video Anomaly Detection with Support Vector Machine (SVM) Algorithm
3.4. Pixel-Based Video Anomaly Detection (PBVAD) Method
3.4.1. Motion Influence Map (MIM) Algorithm
- (i)
- Optical flow;
- (ii)
- Calculation of the effect of movement between blocks;
- (iii)
- Calculation of impact weights between both blocks;
- (iv)
- Calculation of motion ray (direction) weights for each block.
Algorithm 2 Motion Influence Map (MIM) |
3.4.2. Pixel-Based Video Anomaly Detection with Motion Influence Map (MIM) Algorithm
4. Experiments and Results
4.1. General Information on Implementation and Test Environment
4.2. Data Set Selection and Implementation
4.3. Test Results and Anomaly Detection Evaluation
4.4. Comparison of the Results
4.5. Discussion
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Xu, Z.; Zeng, X.; Ji, G.; Sheng, B. Improved Anomaly Detection in Surveillance Videos with Multiple Probabilistic Models Inference. Intell. Autom. Soft Comput. 2022, 31, 1703–1717. [Google Scholar] [CrossRef]
- Yadav, R.K.; Kumar, R. A Survey on Video Anomaly Detection. In Proceedings of the 2022 IEEE Delhi Section Conference (DELCON), New Delhi, India, 11–13 February 2022; pp. 1–5. [Google Scholar] [CrossRef]
- Liu, Y.; Yu, H.; Gong, C.; Chen, Y. A real time expert system for anomaly detection of aerators based on computer vision and surveillance cameras. J. Vis. Commun. Image Represent. 2020, 68, 102767, ISSN 1047-3203. [Google Scholar] [CrossRef]
- Liu, Y.; Liu, J.; Zhu, X.; Wei, D.; Huang, X.; Song, L. Learning Task-Specific Representation for Video Anomaly Detection with Spatial-Temporal Attention. In Proceedings of the ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore, 23–27 May 2022; pp. 2190–2194. [Google Scholar]
- Li, W.; Mahadevan, V.; Vasconcelos, N. Anomaly detection and localization in crowded scenes. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 36, 18–32. [Google Scholar]
- Luo, W.; Liu, W.; Lian, D.; Tang, J.; Duan, L.; Peng, X.; Gao, S. Video Anomaly Detection with Sparse Coding Inspired Deep Neural Networks. IEEE Trans. Pattern Anal. Mach. Intell. 2019, 43, 1070–1084. [Google Scholar] [CrossRef]
- Hou, J.; Wu, X.; Chen, J.; Luo, J.; Jia, Y. Unsupervised Deep Learning of Mid-Level Video Representation for Action Recogni-tion. Proc. AAAI Conf. Artif. Intell. 2018, 32. [Google Scholar] [CrossRef]
- Qiang, Y.; Fei, S.; Jiao, Y. Anomaly Detection Based on Latent Feature Training in Surveillance Scenarios. IEEE Access 2021, 9, 68108–68117. [Google Scholar] [CrossRef]
- Sultani, W.; Chen, C.; Shah, M. Real-World Anomaly Detection in Surveillance Videos. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 6479–6488. [Google Scholar]
- Tian, Y.; Pang, G.; Chen, Y.; Singh, R.; Verjans, J.W.; Carneiro, G. Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 10–17 October 2021; pp. 4955–4966. [Google Scholar] [CrossRef]
- Altamimi, A.B.; Ullah, H. Panic Detection in Crowded Scenes. Eng. Technol. Appl. Sci. Res. 2020, 10, 5412–5418. [Google Scholar] [CrossRef]
- e Souza, M.R.; Maia, H.d.A.; Vieira, M.B.; Pedrini, H. Survey on visual rhythms: A spatio-temporal representation for video sequences. Neurocomputing 2020, 402, 409–422. [Google Scholar] [CrossRef]
- Tokmakov, P.; Alahari, K.; Schmid, C. Learning Video Object Segmentation with Visual Memory. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 4481–4490. [Google Scholar]
- Shao, L.; Cai, Z.; Liu, L.; Lu, K. Performance evaluation of deep feature learning for RGB-D image/video classification. Inf. Sci. 2017, 385–386, 266–283. [Google Scholar] [CrossRef]
- Mu, H.; Sun, R.; Yuan, G.; Wang, Y. Abnormal Human Behavior Detection in Videos: A Review. Inf. Technol. Control. 2021, 50, 522–545. [Google Scholar] [CrossRef]
- Gianchandani, U.; Tirupattur, P.; Shah, M. Weakly-Supervised Spatiotemporal Anomaly Detection; University of Central Florida Center for Research in Computer Vision REU: Orlando, FL, USA, 2019. [Google Scholar]
- Ullah, W.; Ullah, A.; Haq, I.U.; Muhammad, K.; Sajjad, M.; Baik, S.W. CNN features with bi-directional LSTM for real-time anomaly detection in surveillance networks. Multimedia Tools Appl. 2020, 80, 16979–16995. [Google Scholar] [CrossRef]
- Wu, C.; Shao, S.; Tunc, C.; Hariri, S. Video Anomaly Detection using Pre-Trained Deep Convolutional Neural Nets and Context Mining. In Proceedings of the 2020 IEEE/ACS 17th International Conference on Computer Systems and Applica-tions (AICCSA), Antalya, Turkey, 2–5 November 2020; pp. 1–8. [Google Scholar] [CrossRef]
- Luo, W.; Liu, W.; Gao, S. A Revisit of Sparse Coding Based Anomaly Detection in Stacked RNN Framework. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 341–349. [Google Scholar] [CrossRef]
- Xu, K.; Jiang, X.; Sun, T. Anomaly Detection Based on Stacked Sparse Coding with Intraframe Classification Strategy. IEEE Trans. Multimed. 2018, 20, 1062–1074. [Google Scholar] [CrossRef]
- Yu, S.; Wang, C.; Mao, Q.; Li, Y.; Wu, J. Cross-Epoch Learning for Weakly Supervised Anomaly Detection in Surveillance Videos. IEEE Signal Process. Lett. 2021, 28, 2137–2141. [Google Scholar] [CrossRef]
- Zhang, D.; Huang, C.; Liu, C.; Xu, Y. Weakly Supervised Video Anomaly Detection via Transformer-Enabled Temporal Re-lation Learning. IEEE Signal Process. Lett. 2022, 29, 1197–1201. [Google Scholar] [CrossRef]
- Park, C.; Cho, M.; Lee, M.; Lee, S. FastAno: Fast Anomaly Detection via Spatio-temporal Patch Transformation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 3–8 January 2022; pp. 2249–2259. [Google Scholar] [CrossRef]
- Morais, R.; Le, V.; Tran, T.; Saha, B.; Mansour, M.; Venkatesh, S. Learning Regularity in Skeleton Trajectories for Anomaly Detection in Videos. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15–20 June 2019; pp. 11988–11996. [Google Scholar] [CrossRef]
- Nayak, R.; Pati, U.C.; Das, S.K. A comprehensive review on deep learning-based methods for video anomaly detection. Image Vis. Comput. 2020, 106, 104078. [Google Scholar] [CrossRef]
- Park, H.; Noh, J.; Ham, B. Learning Memory-Guided Normality for Anomaly Detection. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13–19 June 2020; pp. 14360–14369. [Google Scholar] [CrossRef]
- Ramachandra, B.; Jones, M.; Vatsavai, R. Learning a distance function with a Siamese network to localize anomalies in vid-eos. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Snowmass, CO, USA, 1–5 March 2020; pp. 2598–2607. [Google Scholar]
- Cheng, K.W.; Chen, Y.T.; Fang, W.H. Video anomaly detection and localization using hierarchical feature representation and Gaussian process regression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 2909–2917. [Google Scholar] [CrossRef]
- Erdinç, A. Anomaly Detection with Sparse Unmixing And Gaussian Mixture Modeling of Hyperspectral Images. Master’s Thesis, Bilkent University, The Graduate School of Engineering and Science, Computer Engineering, Ankara, Turkey, 2015. [Google Scholar]
- Fan, Y.; Wen, G.; Li, D.; Qiu, S.; Levine, M.D.; Xiao, F. Video anomaly detection and localization via Gaussian Mixture Fully Convolutional Variational Autoencoder. Comput. Vis. Image Underst. 2020, 195, 102920, ISSN 1077-3142. [Google Scholar] [CrossRef]
- Calderara, S.; Heinemann, U.; Prati, A.; Cucchiara, R.; Tishby, N. Detecting anomalies in people’s trajectories using spectral graph analysis. Comput. Vis. Image Underst. 2011, 115, 1099–1111, ISSN 1077-3142. [Google Scholar] [CrossRef]
- Markovitz, A.; Sharir, G.; Friedman, I.; Zelnik-Manor, L.; Avidan, S. Graph Embedded Pose Clustering for Anomaly Detection. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13–19 June 2020; pp. 10536–10544. [Google Scholar] [CrossRef]
- Chang, Y.; Tu, Z.; Xie, W.; Yuan, J. Clustering driven deep autoencoder for video anomaly detection. In European Conference on Computer Vision; Springer: Cham, Switzerland, 2020; pp. 329–345. [Google Scholar] [CrossRef]
- Wang, Y.; Dai, B.; Hua, G.; Aston, J.; Wipf, D. Recurrent Variational Autoencoders for Learning Nonlinear Generative Mod-els in the Presence of Outliers. IEEE J. Sel. Top. Signal Process. 2018, 12, 1615–1627. [Google Scholar] [CrossRef]
- Zerrouki, Y.; Harrou, F.; Zerrouki, N.; Dairi, A.; Sun, Y. Desertification Detection Using an Improved Variational Autoen-coder-Based Approach Through ETM-Landsat Satellite Data. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2020, 14, 202–213. [Google Scholar] [CrossRef]
- Gong, D.; Liu, L.; Le, V.; Saha, B.; Mansour, M.R.; Venkatesh, S.; Hengel, A.V.D. Memorizing Normality to Detect Anomaly: Memory-Augmented Deep Autoencoder for Unsupervised Anomaly Detection. In Proceedings of the 2019 IEEE/CVF Inter-national Conference on Computer Vision (ICCV), Seoul, Republic of Korea, 27 October–2 November 2019; pp. 1705–1714. [Google Scholar]
- Wang, X.; Che, Z.; Jiang, B.; Xiao, N.; Yang, K.; Tang, J.; Ye, J.; Wang, J.; Qi, Q. Robust Unsupervised Video Anomaly Detec-tion by Multipath Frame Prediction. IEEE Trans. Neural Networks Learn. Syst. 2021, 33, 2301–2312. [Google Scholar] [CrossRef]
- Feng, J.C.; Hong, F.T.; Zheng, W.S. MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021; pp. 14004–14013. [Google Scholar] [CrossRef]
- Georgescu, M.I.; Barbalau, A.; Ionescu, R.T.; Khan, F.S.; Popescu, M.; Shah, M. Anomaly detection in video via self-supervised and multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 12742–12752. [Google Scholar] [CrossRef]
- Huang, C.; Wen, J.; Xu, Y.; Jiang, Q.; Yang, J.; Wang, Y.; Zhang, D. Self-Supervised Attentive Generative Adversarial Net-works for Video Anomaly Detection. IEEE Trans. Neural Networks Learn. Syst. 2022, 34, 9389–9403. [Google Scholar] [CrossRef] [PubMed]
- Zaheer, M.Z.; Mahmood, A.; Shin, H.; Lee, S.-I. A Self-Reasoning Framework for Anomaly Detection Using Video-Level Labels. IEEE Signal Process. Lett. 2020, 27, 1705–1709. [Google Scholar] [CrossRef]
- Muhr, D.; Affenzeller, M.; Küng, J. A Probabilistic Transformation of Distance-Based Outliers. Mach. Learn. Knowl. Extr. 2023, 5, 782–802. [Google Scholar] [CrossRef]
- Adam, A.; Rivlin, E.; Shimshoni, I.; Reinitz, D. Robust Real-Time Unusual Event Detection using Multiple Fixed-Location Monitors. IEEE Trans. Pattern Anal. Mach. Intell. 2008, 30, 555–560. [Google Scholar] [CrossRef] [PubMed]
- Hao, Y.; Xu, Z.-J.; Liu, Y.; Wang, J.; Fan, J.-L. Effective Crowd Anomaly Detection Through Spatio-temporal Texture Analy-sis. Int. J. Autom. Comput. 2018, 16, 27–39. [Google Scholar] [CrossRef]
- Le, V.-T.; Kim, Y.-G. Attention-based residual autoencoder for video anomaly detection. Appl. Intell. 2022, 53, 3240–3254. [Google Scholar] [CrossRef]
- Ravanbakhsh, M.; Nabi, M.; Mousavi, H.; Sangineto, E.; Sebe, N. Plug-and-Play CNN for Crowd Motion Analysis: An Ap-plication in Abnormal Event Detection. In Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA, 12–15 March 2018; pp. 1689–1698. [Google Scholar] [CrossRef]
- Sabokrou, M.; Fayyaz, M.; Fathy, M.; Moayed, Z.; Klette, R. Deep-anomaly: Fully convolutional neural network for fast anomaly detection in crowded scenes. Comput. Vis. Image Underst. 2018, 172, 88–97, ISSN 1077-3142. [Google Scholar] [CrossRef]
- Smeureanu, S.; Ionescu, R.T. Real-Time Deep Learning Method for Abandoned Luggage Detection in Video. In Proceedings of the 2018 26th European Signal Processing Conference (EUSIPCO), Rome, Italy, 3–7 September 2018; pp. 1775–1779. [Google Scholar] [CrossRef]
- Xu, D.; Song, R.; Wu, X.; Li, N.; Feng, W.; Qian, H. Video anomaly detection based on a hierarchical activity discovery with-in spatio-temporal contexts. Neurocomputing 2014, 143, 144–152. [Google Scholar] [CrossRef]
- Mehran, R.; Oyama, A.; Shah, M. Abnormal Crowd Behavior Detection using Social Force Model. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 935–942. [Google Scholar] [CrossRef]
- Lu, C.; Shi, J.; Jia, J. Abnormal Event Detection at 150 FPS in MATLAB. In Proceedings of the 2013 IEEE International Con-ference on Computer Vision, Sydney, NSW, Australia, 1–8 December 2013; pp. 2720–2727. [Google Scholar]
- Liu, W.; Luo, W.; Lian, D.; Gao, S. Future Frame Prediction for Anomaly Detection-A New Baseline. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 6536–6545. [Google Scholar]
- Wu, P.; Liu, J.; Shi, Y.; Sun, Y.; Shao, F.; Wu, Z.; Yang, Z. Not only look, but also listen: Learning multimodal violence detec-tion under weak supervision. In Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, 23–28 August 2020; Springer International Publishing: Cham, Switzerland; pp. 322–339, Part XXX 16. [Google Scholar]
- Xia, X.; Gao, Y. Video Abnormal Event Detection Based on One-Class Neural Network. Comput. Intell. Neurosci. 2021, 2021, 1955116. [Google Scholar] [CrossRef]
- Amin, S.U.; Ullah, M.; Sajjad, M.; Cheikh, F.A.; Hijji, M.; Hijji, A.; Muhammad, K. EADN: An Efficient Deep Learning Model for Anomaly Detection in Videos. Mathematics 2022, 10, 1555. [Google Scholar] [CrossRef]
- Öngün, C. Anomaly Detection for Video Surveillance in Crowded Environments. Master’s Thesis, Middle East Technical University, Electrical and Electronics Engineering, Ankara, Turkey, 2014. [Google Scholar]
- Bertini, M.; Del Bimbo, A.; Seidenari, L. Multi-scale and real-time non-parametric approach for anomaly detection and lo-calization. Comput. Vis. Image Underst. 2011, 116, 320–329, ISSN 1077-3142. [Google Scholar] [CrossRef]
- Jiang, F.; Yuan, J.; Tsaftaris, S.A.; Katsaggelos, A.K. Anomalous video event detection using spatiotemporal context. Comput. Vis. Image Underst. 2011, 115, 323–333, ISSN 1077-3142. [Google Scholar] [CrossRef]
- Li, N.; Wu, X.; Xu, D.; Guo, H.; Feng, W. Spatio-temporal context analysis within video volumes for anomalous-event detec-tion and localization. Neurocomputing 2015, 155, 309–319. [Google Scholar] [CrossRef]
- Wang, X.; Huang, Z.; Liao, B.; Huang, L.; Gong, Y.; Huang, C. Real-time and accurate object detection in compressed video by long short-term feature aggregation. Comput. Vis. Image Underst. 2021, 206, 103188, ISSN 1077-3142. [Google Scholar] [CrossRef]
- Georgescu, M.I.; Ionescu, R.; Khan, F.S.; Popescu, M.; Shah, M. A Background-Agnostic Framework with Adversarial Train-ing for Abnormal Event Detection in Video. IEEE Trans. Pattern Anal. Mach. Intell. 2022, 44, 4505–4523. [Google Scholar] [CrossRef]
- Ravanbakhsh, M.; Nabi, M.; Sangineto, E.; Marcenaro, L.; Regazzoni, C.S.; Sebe, N. Abnormal event detection in videos us-ing generative adversarial nets. In Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China, 17–20 September 2017; pp. 1577–1581. [Google Scholar] [CrossRef]
- Zaheer, M.Z.; Mahmood, A.; Astrid, M.; Lee, S.I. Claws: Clustering assisted weakly supervised learning with normalcy sup-pression for anomalous event detection. In Proceedings of the European Conference on Computer Vision, online, 23–28 August 2020; Springer: Cham, Switzerland, 2020; pp. 358–376. [Google Scholar]
- Sikdar, A.; Chowdhury, A.S. An adaptive training-less framework for anomaly detection in crowd scenes. Neurocomputing 2020, 415, 317–331. [Google Scholar] [CrossRef]
- Tudor Ionescu, R.; Smeureanu, S.; Alexe, B.; Popescu, M. Unmasking the abnormal events in video. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2895–2903. [Google Scholar] [CrossRef]
- Xu, D.; Yan, Y.; Ricci, E.; Sebe, N. Detecting anomalous events in videos by learning deep representations of appearance and motion. Comput. Vis. Image Underst. 2017, 156, 117–127. [Google Scholar] [CrossRef]
- Sarikan, S.S.; Ozbayoglu, A.M. Anomaly Detection in Vehicle Traffic with Image Processing and Machine Learning. Procedia Comput. Sci. 2018, 140, 64–69. [Google Scholar] [CrossRef]
- Khan, M.U.K.; Park, H.-S.; Kyung, C.-M. Rejecting Motion Outliers for Efficient Crowd Anomaly Detection. IEEE Trans. Inf. Forensics Secur. 2018, 14, 541–556. [Google Scholar] [CrossRef]
- Kiranyaz, S.; Zabihi, M.; Rad, A.B.; Ince, T.; Hamila, R.; Gabbouj, M. Real-time phonocardiogram anomaly detection by adaptive 1D Convolutional Neural Networks. Neurocomputing 2020, 411, 291–301. [Google Scholar] [CrossRef]
- Ajlouni, N.; Özyavaş, A.; Takaoğlu, M.; Takaoğlu, F.; Ajlouni, F. Medical image diagnosis based on adaptive Hybrid Quan-tum CNN. BMC Med. Imaging 2023, 23, 126. [Google Scholar] [CrossRef] [PubMed]
- Duong, H.-T.; Le, V.-T.; Hoang, V.T. Deep Learning-Based Anomaly Detection in Video Surveillance: A Survey. Sensors 2023, 23, 5024. [Google Scholar] [CrossRef] [PubMed]
- Mehta, R.; Sharifzadeh, S.; Palade, V.; Tan, B.; Daneshkhah, A.; Karayaneva, Y. Deep Learning Techniques for Radar-Based Continuous Human Activity Recognition. Mach. Learn. Knowl. Extr. 2023, 5, 1493–1518. [Google Scholar] [CrossRef]
- Ionescu, R.T.; Khan, F.S.; Georgescu, M.I.; Shao, L. Object-Centric Auto-Encoders and Dummy Anomalies for Abnormal Event Detection in Video. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15–20 June 2019; pp. 7834–7843. [Google Scholar] [CrossRef]
- Perazzi, F.; Khoreva, A.; Benenson, R.; Schiele, B.; Sorkine-Hornung, A. Learning Video Object Segmentation from Static Images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 2663–2672. [Google Scholar]
- Yang, C.L.; Wu, T.H.; Lai, S.H. Moving-object-aware anomaly detection in surveillance videos. In Proceedings of the 2021 17th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Washington, DC, USA, 16–19 November 2021; pp. 1–8. [Google Scholar] [CrossRef]
- Liu, T.; Zhang, C.; Wang, L. Integrated Multiscale Appearance Features and Motion Information Prediction Network for Anomaly Detection. Comput. Intell. Neurosci. 2021, 2021, 6789956. [Google Scholar] [CrossRef]
- Nguyen, T.N.; Meunier, J. Anomaly Detection in Video Sequence with Appearance-Motion Correspondence. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea, 27 October–2 November 2019; pp. 1273–1283. [Google Scholar] [CrossRef]
- Zhang, X.; Yang, S.; Zhang, J.; Zhang, W. Video anomaly detection and localization using motion-field shape description and homogeneity testing. Pattern Recognit. 2020, 105, 107394, ISSN 0031-3203. [Google Scholar] [CrossRef]
- Zhu, Y.; Newsam, S. Motion-aware feature for improved video anomaly detection. arXiv 2019, arXiv:1907.10211. [Google Scholar]
- Rehman, A.U.; Tariq, S.; Farooq, H.; Jaleel, A.; Wasif, S.M. Anomaly Detection with Particle Filtering for Online Video Sur-veillance. IEEE Access 2021, 9, 19457–19468. [Google Scholar] [CrossRef]
- Leyva, R.; Sanchez, V.; Li, C.-T. Video Anomaly Detection with Compact Feature Sets for Online Performance. IEEE Trans. Image Process. 2017, 26, 3463–3478. [Google Scholar] [CrossRef]
- Thakare, K.V.; Sharma, N.; Dogra, D.P.; Choi, H.; Kim, I.-J. A multi-stream deep neural network with late fuzzy fusion for real-world anomaly detection. Expert Syst. Appl. 2022, 201, 117030. [Google Scholar] [CrossRef]
- Wang, B.; Yang, C.; Chen, Y. Detection Anomaly in Video Based on Deep Support Vector Data Description. Comput. Intell. Neurosci. 2022, 2022, 5362093. [Google Scholar] [CrossRef]
- Wang, Z.; Zhang, Y.; Wang, G.; Xie, P. Main-Auxiliary Aggregation Strategy for Video Anomaly Detection. IEEE Signal Process. Lett. 2021, 28, 1794–1798. [Google Scholar] [CrossRef]
- Santos, F.P.; Ribeiro, L.S.; Ponti, M.A. Generalization of feature embeddings transferred from different video anomaly detection domains. J. Vis. Commun. Image Represent. 2019, 60, 407–416. [Google Scholar] [CrossRef]
- Ganokratanaa, T.; Aramvith, S.; Sebe, N. Unsupervised Anomaly Detection and Localization Based on Deep Spati-otemporal Translation Network. IEEE Access 2020, 8, 50312–50329. [Google Scholar] [CrossRef]
- Lee, D.-G.; Suk, H.-I.; Park, S.-K.; Lee, S.-W. Motion Influence Map for Unusual Human Activity Detection and Localization in Crowded Scenes. IEEE Trans. Circuits Syst. Video Technol. 2015, 25, 1612–1623. [Google Scholar] [CrossRef]
Data Set | File | Frame | Video Length |
---|---|---|---|
Abuse | Abuse002_x264.mp4 | 865 | 00:28 s |
Assault | Assault002_x264.mp4 | 2523 | 01:24 s |
Burglary | Burglary012_x264.mp4 | 1698 | 00:56 s |
Explosion | Explosion045_x264.mp4 | 757 | 00:25 s |
Fighting | Fighting006_x264.mp4 | 944 | 00:31 s |
Road accidents | Road Accidents002_x264.mp4 | 347 | 00:11 s |
Data Set | FBVAD-kNN AUC (%) | FBVAD-SVM AUC (%) | PBVAD-MIM AUC (%) |
---|---|---|---|
Abuse | 98.80 | 77.60 | 74.20 |
Assault | 97.90 | 75.13 | 86.60 |
Burglary | 96.60 | 75.11 | 75.80 |
Explosion | 99.30 | 78.70 | 73.10 |
Fighting | 96.80 | 77.20 | 78.80 |
Road accidents | 98.30 | 80.10 | 95.80 |
Data Set | AUC | Precision | Sensitivity | F-Score |
---|---|---|---|---|
Abuse | 98.80 | 0.998 | 0.9762 | 0.974 |
Assault | 97.90 | 0.996 | 0.9569 | 0.955 |
Burglary | 96.60 | 0.993 | 0.9325 | 0.928 |
Explosion | 99.30 | 0.999 | 0.9875 | 0.984 |
Fighting | 96.80 | 0.994 | 0.9372 | 0.925 |
Road accidents | 98.30 | 0.998 | 0.9565 | 0.955 |
Data Set | AUC | Precision | Sensitivity | F Score |
---|---|---|---|---|
Abuse | 77.60 | 0.998 | 0.7757 | 0.874 |
Assault | 75.13 | 0.992 | 0.7515 | 0.857 |
Burglary | 75.11 | 0.992 | 0.7520 | 0.858 |
Explosion | 78.70 | 0.995 | 0.7873 | 0.881 |
Fighting | 77.20 | 0.994 | 0.7722 | 0.871 |
Road accidents | 80.10 | 0.996 | 0.8012 | 0.890 |
Data Set | AUC | Precision | Sensitivity | F Score |
---|---|---|---|---|
Abuse | 74.20 | 0.484 | 0.7419 | 0.652 |
Assault | 86.60 | 0.731 | 0.8657 | 0.845 |
Burglary | 75.80 | 0.516 | 0.7581 | 0.681 |
Explosion | 73.10 | 0.462 | 0.7308 | 0.632 |
Fighting | 78.80 | 0.576 | 0.7879 | 0.731 |
Road accidents | 95.80 | 0.917 | 0.9583 | 0.957 |
Data Set | Samples | Actual Positives | Actual Negatives | True Positives | True Negatives | False Negatives |
---|---|---|---|---|---|---|
Abuse | 865 | 194 | 671 | 184 | 671 | 10 |
Assault | 2523 | 627 | 1896 | 573 | 1896 | 54 |
Burglary | 1698 | 422 | 1276 | 365 | 1276 | 57 |
Explosion | 757 | 161 | 596 | 156 | 596 | 5 |
Fighting | 944 | 215 | 729 | 185 | 729 | 30 |
Road accidents | 347 | 69 | 278 | 63 | 278 | 6 |
Data Set | Samples | Actual Positives | Actual Negatives | True Positives | False Negatives |
---|---|---|---|---|---|
Abuse | 865 | 865 | 0 | 671 | 194 |
Assault | 2523 | 2523 | 0 | 1896 | 627 |
Burglary | 1698 | 1698 | 0 | 1276 | 422 |
Explosion | 757 | 757 | 0 | 596 | 161 |
Fighting | 944 | 944 | 0 | 729 | 215 |
Road accidents | 347 | 347 | 0 | 278 | 69 |
Data Set | Samples | Actual Positives | Actual Negatives | True Positives | True Negatives | False Negatives |
---|---|---|---|---|---|---|
Abuse | 865 | 31 | 834 | 15 | 834 | 16 |
Assault | 2523 | 67 | 2456 | 49 | 2456 | 18 |
Burglary | 1698 | 31 | 1667 | 16 | 1667 | 15 |
Explosion | 757 | 26 | 731 | 12 | 731 | 14 |
Fighting | 944 | 33 | 911 | 19 | 911 | 14 |
Road accidents | 347 | 12 | 335 | 11 | 335 | 1 |
Data Set | Frame Count | Video Length (min:s) | FBVAD -kNN (min:s) | FBVAD -SVM (min:s) | PBVAD -MIM (min:s) |
---|---|---|---|---|---|
Abuse | 865 | 00:28 | 00:40 | 00:40 | 00:40 |
Assault | 2523 | 01:24 | 02:12 | 02:12 | 02:12 |
Burglary | 1698 | 00:56 | 01:05 | 01:05 | 00:43 |
Explosion | 757 | 00:25 | 00:36 | 00:36 | 00:29 |
Fighting | 944 | 00:31 | 00:44 | 00:44 | 00:43 |
Road accidents | 347 | 00:11 | 00:13 | 00:13 | 00:17 |
Data Set | Normality Frame | Anomaly Frame-I | Anomaly Frame-II |
---|---|---|---|
Sample-1: Burglary018 | Normality Frame No.: 16–130 | Anomaly/Burglary Frame No.: 232 | Anomaly/Burglary Frame No.: 510 |
Sample-2: Abuse002 | Normality Frame No.: 1–150 | Anomaly/Abuse Frame No.: 420 | Anomaly/Abuse Frame No.: 630 |
Sample-3: Assault025 | Normality Frame No.: 1–220 | Anomaly/Assault Frame No.: 450 | Anomaly/Assault Frame No.: 1770 |
Sample-4: Road Accidents002 | Normality Frame No.: 1–220 | Anomaly/Road Accident Frame No.: 270 | Anomaly/Road Accident Frame No.: 330 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Tutar, H.; Güneş, A.; Zontul, M.; Aslan, Z. A Hybrid Approach to Improve the Video Anomaly Detection Performance of Pixel- and Frame-Based Techniques Using Machine Learning Algorithms. Computation 2024, 12, 19. https://doi.org/10.3390/computation12020019
Tutar H, Güneş A, Zontul M, Aslan Z. A Hybrid Approach to Improve the Video Anomaly Detection Performance of Pixel- and Frame-Based Techniques Using Machine Learning Algorithms. Computation. 2024; 12(2):19. https://doi.org/10.3390/computation12020019
Chicago/Turabian StyleTutar, Hayati, Ali Güneş, Metin Zontul, and Zafer Aslan. 2024. "A Hybrid Approach to Improve the Video Anomaly Detection Performance of Pixel- and Frame-Based Techniques Using Machine Learning Algorithms" Computation 12, no. 2: 19. https://doi.org/10.3390/computation12020019