Edge-Computing-Enabled Abnormal Activity Recognition for Visual Surveillance
Abstract
:1. Introduction
2. Related Work
2.1. Visual Analytics and Surveillance Systems
2.2. Edge Computing for Visual Surveillance
2.3. Deep Learning Models for Video Classification
2.3.1. Convolutional 3D (C3D)
2.3.2. Recurrent Neural Network (RNN)
2.3.3. Bidirectional LSTM (Bi-LSTM)
3. Proposed System and Methodology
3.1. Proposed Deep Learning Architecture and Methodology for Anomaly Detection
3.2. Edge Deployment of the Model
4. Experimental Results
4.1. Experimental Setup
4.2. Description of the Datasets Used
4.3. Results
4.4. Comparative Performance Analysis and Discussion
4.5. Limitations of the Current Work and Future Scope
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Sharma, L.; Lohan, N. Performance analysis of moving object detection using BGS techniques in visual surveillance. Int. J. Spatio-Temporal Data Sci. 2019, 1, 22–53. [Google Scholar] [CrossRef]
- Tripathi, R.K.; Jalal, A.S.; Agrawal, S.C. Abandoned or removed object detection from visual surveillance: A review. Multimed. Tools Appl. 2019, 78, 7585–7620. [Google Scholar] [CrossRef]
- Gawande, U.; Hajari, K.; Golhar, Y. Pedestrian detection and tracking in video surveillance system: Issues, comprehensive review, and challenges. In Recent Trends in Computational Intelligence; Intechopen: London, UK, 2020; pp. 1–24. [Google Scholar]
- Gundogdu, E.; Parıldı, E.S.; Solmaz, B.; Yücesoy, V.; Koç, A. Deep learning-based fine-grained car make/model classification for visual surveillance. In Proceedings of the Counterterrorism, Crime Fighting, Forensics, and Surveillance Technologies, Warsaw, Poland, 11–12 September 2017; SPIE: Bellingham, WA, USA, 2017; Volume 10441, pp. 179–184. [Google Scholar]
- Zitouni, M.S.; Sluzek, A.; Bhaskar, H. Towards understanding socio-cognitive behaviors of crowds from visual surveillance data. Multimed. Tools Appl. 2020, 79, 1781–1799. [Google Scholar] [CrossRef]
- Santhosh, K.K.; Dogra, D.P.; Roy, P.P. Anomaly detection in road traffic using visual surveillance: A survey. ACM Comput. Surv. (CSUR) 2020, 53, 1–26. [Google Scholar] [CrossRef]
- Ansari, M.A.; Singh, D.K. An expert video surveillance system to identify and mitigate shoplifting in megastores. Multimed. Tools Appl. 2022, 81, 22497–22525. [Google Scholar] [CrossRef]
- Wu, Z.; Yao, T.; Fu, Y.; Jiang, Y.G. Deep learning for video classification and captioning. In Frontiers of Multimedia Research; ACM: New York, NY, USA, 2017; pp. 3–29. [Google Scholar]
- Mliki, H.; Bouhlel, F.; Hammami, M. Human activity recognition from UAV-captured video sequences. Pattern Recognit. 2020, 100, 107140. [Google Scholar] [CrossRef]
- Sunil, A.; Sheth, M.H.; Shreyas, E. Usual and unusual human activity recognition in video using deep learning and artificial intelligence for security applications. In Proceedings of the 2021 Fourth International Conference on Electrical, Computer and Communication Technologies (ICECCT), Erode, India, 15–17 September 2021; IEEE: Piscataway Township, NJ, USA, 2021; pp. 1–6. [Google Scholar]
- Wassim, A. Abnormal Activity Detection In Crowded Scenes Using Video Surveillance. In Proceedings of the Cyber-Physical Systems and Control, Sydney, Australia, 21–25 April 2020; pp. 106–118. [Google Scholar]
- Nawaratne, R.; Alahakoon, D.; De Silva, D.; Yu, X. Spatiotemporal anomaly detection using deep learning for real-time video surveillance. IEEE Trans. Ind. Inform. 2019, 16, 393–402. [Google Scholar] [CrossRef]
- Zhou, J.T.; Du, J.; Zhu, H.; Peng, X.; Liu, Y.; Goh, R.S.M. Anomalynet: An anomaly detection network for video surveillance. IEEE Trans. Inf. Forensics Secur. 2019, 14, 2537–2550. [Google Scholar] [CrossRef]
- Rodrigues, R.; Bhargava, N.; Velmurugan, R.; Chaudhuri, S. Multi-timescale trajectory prediction for abnormal human activity detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Snowmass Village, CO, USA, 1–5 March 2020; pp. 2626–2634. [Google Scholar]
- Fan, Z.; Yin, J.; Song, Y.; Liu, Z. Real-time and accurate abnormal behavior detection in videos. Mach. Vis. Appl. 2020, 31, 72. [Google Scholar] [CrossRef]
- Ullah, W.; Ullah, A.; Haq, I.U.; Muhammad, K.; Sajjad, M.; Baik, S.W. CNN features with bi-directional LSTM for real-time anomaly detection in surveillance networks. Multimed. Tools Appl. 2021, 80, 16979–16995. [Google Scholar] [CrossRef]
- Singh, T.; Vishwakarma, D.K. A deeply coupled ConvNet for human activity recognition using dynamic and RGB images. Neural Comput. Appl. 2021, 33, 469–485. [Google Scholar] [CrossRef]
- Shreyas, D.; Raksha, S.; Prasad, B. Implementation of an anomalous human activity recognition system. SN Comput. Sci. 2020, 1, 168. [Google Scholar] [CrossRef]
- Beddiar, D.R.; Nini, B.; Sabokrou, M.; Hadid, A. Vision-based human activity recognition: A survey. Multimed. Tools Appl. 2020, 79, 30509–30555. [Google Scholar] [CrossRef]
- Pareek, P.; Thakkar, A. A survey on video-based human action recognition: Recent updates, datasets, challenges, and applications. Artif. Intell. Rev. 2021, 54, 2259–2322. [Google Scholar] [CrossRef]
- Kumar, S.; Kumar, S.; Raman, B.; Sukavanam, N. Human action recognition in a wide and complex environment. In Proceedings of the Real-Time Image and Video Processing 2011, San Francisco, CA, USA, 1 January 2011; SPIE: Bellingham, WA, USA, 2011; Volume 7871, pp. 176–187. [Google Scholar]
- Kumar, S.; Kumar, S.; Sukavanam, N.; Raman, B. Human visual system and segment-based disparity estimation. AEU-Int. J. Electron. Commun. 2013, 67, 372–381. [Google Scholar] [CrossRef]
- Kumar, S.; Kumar, S.; Sukavanam, N.; Raman, B. Dual tree fractional quaternion wavelet transform for disparity estimation. ISA Trans. 2014, 53, 547–559. [Google Scholar] [CrossRef]
- Cob-Parro, A.C.; Losada-Gutiérrez, C.; Marrón-Romera, M.; Gardel-Vicente, A.; Bravo-Muñoz, I. Smart video surveillance system based on edge computing. Sensors 2021, 21, 2958. [Google Scholar] [CrossRef]
- Zhang, J.; Xu, C.; Gao, Z.; Rodrigues, J.J.; de Albuquerque, V.H.C. Industrial pervasive edge computing-based intelligence IoT for surveillance saliency detection. IEEE Trans. Ind. Inform. 2020, 17, 5012–5020. [Google Scholar] [CrossRef]
- Rajavel, R.; Ravichandran, S.K.; Harimoorthy, K.; Nagappan, P.; Gobichettipalayam, K.R. IoT-based smart healthcare video surveillance system using edge computing. J. Ambient. Intell. Humaniz. Comput. 2022, 13, 3195–3207. [Google Scholar] [CrossRef]
- Ahmed, I.; Ahmad, M.; Rodrigues, J.J.; Jeon, G. Edge computing-based person detection system for top view surveillance: Using CenterNet with transfer learning. Appl. Soft Comput. 2021, 107, 107489. [Google Scholar] [CrossRef]
- Yang, B.; Cao, X.; Yuen, C.; Qian, L. Offloading optimization in edge computing for deep-learning-enabled target tracking by internet of UAVs. IEEE Internet Things J. 2020, 8, 9878–9893. [Google Scholar] [CrossRef]
- Kumar, P.P.; Pal, A.; Kant, K. Resource efficient edge computing infrastructure for video surveillance. IEEE Trans. Sustain. Comput. 2021, 7, 774–785. [Google Scholar] [CrossRef]
- Ananthanarayanan, G.; Bahl, P.; Bodík, P.; Chintalapudi, K.; Philipose, M.; Ravindranath, L.; Sinha, S. Real-time video analytics: The killer app for edge computing. Computer 2017, 50, 58–67. [Google Scholar] [CrossRef]
- Hussain, T.; Muhammad, K.; Ullah, A.; Del Ser, J.; Gandomi, A.H.; Sajjad, M.; Baik, S.W.; de Albuquerque, V.H.C. Multiview summarization and activity recognition meet edge computing in IoT environments. IEEE Internet Things J. 2020, 8, 9634–9644. [Google Scholar] [CrossRef]
- Aishwarya, D.; Minu, R. Edge computing based surveillance framework for real time activity recognition. ICT Express 2021, 7, 182–186. [Google Scholar]
- Subramanian, R.R.; Vasudevan, V. A deep genetic algorithm for human activity recognition leveraging fog computing frameworks. J. Vis. Commun. Image Represent. 2021, 77, 103132. [Google Scholar] [CrossRef]
- Tran, D.; Bourdev, L.; Fergus, R.; Torresani, L.; Paluri, M. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 4489–4497. [Google Scholar]
- Sultani, W.; Chen, C.; Shah, M. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 6479–6488. [Google Scholar]
- Landi, F.; Snoek, C.G.; Cucchiara, R. Anomaly locality in video surveillance. arXiv 2019, arXiv:1901.10364. [Google Scholar]
- UCF Dataset. Real-world Anomaly Detection in Surveillance Videos. Available online: https://www.crcv.ucf.edu/projects/real-world/ (accessed on 3 October 2022).
- Amanatidis, P.; Iosifidis, G.; Karampatzakis, D. Comparative Evaluation of Machine Learning Inference Machines on Edge-class Devices. In Proceedings of the 25th Pan-Hellenic Conference on Informatics, Volos, Greece, 26–28 November 2021; pp. 102–106. [Google Scholar]
References | Year | Features | Hardware | Algorithm | Dataset |
---|---|---|---|---|---|
Cob-Parro et al. [24] | 2021 | Human detection and classification | UpSquared2 system, Intel Myriad X | MobileNetSSD Model | EPFL dataset |
Zhang et al. [25] | 2020 | Surveillance saliency detection | DAVIS, UVSD | ||
Rajavel et al. [26] | 2022 | Patient surveillance, fall detection | Raspberry Pi 3, IP Cameras | Four-layered IoT architecture including sensors, processing, and cloud | Auvinet (2010), DIRO-Université de Montréal Dataset |
Ahmed et al. [27] | 2021 | Person Detection, Edge Computing | Edge camera, VPU, local server, cloud | One stage deep learning-based person detector- CenterNet | Self-recorded dataset 2k images |
Yang et al. [28] | 2020 | Vehicle detection and tracking | UAV, camera, MEC | CNN architecture, hierarchical ML tasks distribution (HMTD) framework | ImageNet |
Pradeepkumar et al. [29] | 2021 | Vehicular traffic monitoring | Camera, edge node | YLLO object detector, BATS bandwidth optimizer algorithms | MS-COCO, UA-Detrac |
Ananthanarayanan et al. [30] | 2017 | Abnormal traffic patterns | Steerable Cameras | ‘Rocket’ software stack, DNN | Crowd-sourced videos |
Hussain et al. [31] | 2020 | Multiview video summarization, activity recognition | Movidius NCS, vision sensor, wireless sensor | Lightweight CNN, autoencoders, SVM | MVS Office, UCF-50, and YouTube 11 Datasets |
Aishwarya et al. [32] | 2021 | Normal and abnormal activity recognition in indoor environment | Raspberry Pi, Camera | CNN, Background Segmentation | Own dataset (SRMIT) |
Subramanian et al. [33] | 2021 | Fog+Activity Detection | Genetic Algorithm + Background Segmentation | Holllywood2, UCF-ARG, KTH |
Anomaly | Videos |
---|---|
Abuse | 50 |
Arrest | 50 |
Arson | 50 |
Assault | 50 |
Burglary | 100 |
Explosion | 50 |
Fighting | 50 |
Road Accidents | 150 |
Robbery | 150 |
Shooting | 50 |
Shoplifting | 50 |
Stealing | 100 |
Vandalism | 50 |
Normal Videos | 950 |
Event | Precision | Recall | F1 Score |
---|---|---|---|
Arrest | 0.79 | 0.79 | 0.79 |
Arson | 0.79 | 0.78 | 0.78 |
Assault | 0.78 | 0.78 | 0.78 |
Burglary | 0.79 | 0.79 | 0.79 |
Explosion | 0.81 | 0.81 | 0.81 |
Hitting | 0.79 | 0.79 | 0.79 |
Road Accidents | 0.81 | 0.79 | 0.80 |
Robbery | 0.78 | 0.78 | 0.78 |
Shooting | 0.77 | 0.79 | 0.78 |
Shoplifting | 0.77 | 0.79 | 0.78 |
Stealing | 0.78 | 0.78 | 0.78 |
Vandalism | 0.79 | 0.80 | 0.79 |
Normal Events | 0.79 | 0.81 | 0.80 |
Model | Training Accuracy | Test Accuracy |
---|---|---|
C3D | 55.16% | 45.87% |
RNN | 85.61% | 59.34% |
Bi-LSTM | 66.46% | 60.96% |
Proposed Model | 91.62% | 80.92% |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Ali, M.; Goyal, L.; Sharma, C.M.; Kumar, S. Edge-Computing-Enabled Abnormal Activity Recognition for Visual Surveillance. Electronics 2024, 13, 251. https://doi.org/10.3390/electronics13020251
Ali M, Goyal L, Sharma CM, Kumar S. Edge-Computing-Enabled Abnormal Activity Recognition for Visual Surveillance. Electronics. 2024; 13(2):251. https://doi.org/10.3390/electronics13020251
Chicago/Turabian StyleAli, Musrrat, Lakshay Goyal, Chandra Mani Sharma, and Sanoj Kumar. 2024. "Edge-Computing-Enabled Abnormal Activity Recognition for Visual Surveillance" Electronics 13, no. 2: 251. https://doi.org/10.3390/electronics13020251