Towards an Indoor Gunshot Detection and Notification System Using Deep Learning
Abstract
:1. Introduction
- The United States sees the most school shootings in the world [3]. Shootings inside other indoor places such as in homes, shopping malls, clubs, and places of worship are also becoming widespread around the world [4,5]. The proposed device can be attached to the walls or ceilings of these places—similar to smoke detectors—and they can notify the police as soon as a gunshot is fired. The proposed system will help to stop the shooter early and the injured people can be taken to the hospital quickly, thus more lives can be saved.
- Individuals affected by gun violence, including witnesses, bystanders, and neighbors, may undergo feelings of stress, depression, anxiety, and post-traumatic stress disorder (PTSD). More than 5% of America’s children have witnessed a shooting and it causes them psychological distress [6]. The proposed device can lead the police to the crime scene as soon as possible and can give peace of mind.
- Solely in 2010, emergency rooms received 36,000 firearm assault victims, with 25,000 requiring hospital admission, resulting in a staggering USD 630 million in medical expenses. The overall economic impact of gun violence on the American economy is estimated to be at least USD 229 billion annually [6]. The proposed device will bring the police quickly and reduce medical costs and property damage.
2. Related Work
3. Materials and Methods
3.1. A Deep Learning Model for Gunshot Detection
3.1.1. Dataset Generation
3.1.2. Normalization and Feature Extraction
3.1.3. Convolutional Neural Network Architecture
- MFCC Input Image: The MFCC image is structured as a tensor with dimensions of (32, 32, 1). To ensure compatibility with deep learning models, the pixel data type undergoes conversion to a floating-point format. For the purpose of pixel value normalization, we calculate and store the mean and standard deviation of the training dataset in a separate file. Subsequently, each pixel value in all dataset images is normalized by subtracting the mean and dividing by the standard deviation.
- The Convolutional Layer: A 2-D convolutional layer applies sliding convolutional filters to the input data. It conducts convolution by sliding these filters along both the vertical and horizontal axes, computing the dot product between the weights and input, and then adding a bias term [39]. The proposed model incorporates two convolutional layers, each utilizing 3 × 3 filters. These filters are initialized with random values and function as learnable network parameters. For example, in Figure 3, the conv2d layer comprises 2 filters, each sized 3 × 3 with padding, resulting in 2 output layers having the same height and width as the input layer. Similarly, the conv2d_1 layer includes 2 filters, each sized 3 × 3 with padding.
- The Activation Layer: Non-linear activation functions, specifically the rectified linear unit (ReLU) [40], are applied after the convolutional and dense layers (except the last dense layer). The ReLU layer performs a threshold operation on each element, setting any value less than zero to zero. This activation function introduces non-linearity into the network, enabling it to capture intricate patterns and enhance its classification performance.
- The Max Pooling Layer: This layer conducts down-sampling by partitioning. Here, the input is divided into 2 × 2 size rectangular regions and the maximum number from each region is sampled [41].
- The Flatten Layer: The spatial dimensions of the input are collapsed here, and it converts it into a one-dimensional vector. It transforms the input tensor having dimensions of (8, 8, 1) into a 64 size single-dimensional vector.
- The Dense Layer: The dot product between the input and a weight matrix is computed in this layer. Following this, a bias vector is added to the result [42,43]. Random values are used to initialize both the weight matrix and bias, and theyserve as learnable parameters of the model. This layer is also known as the fully connected (FC) layer.
- Loss Function and Optimizer: The last fully connected layer, dense_2, integrates the extracted features for image classification. Consequently, the last dense layer output size is set to one for binary classification. Then a score function, Sigmoid [44], is used. The agreement between predicted scores and the ground truth labels are quantified by the loss function. The job of the optimizer is to find the global minima by varying the network parameters. Global minima can be achieved when the loss function reaches its minimum value. In the proposed model, the binary_crossentropy loss is computed, and the RMSprop [45] optimizer is utilized.
3.1.4. Training the Deep Learning Model
3.2. Prototype System Architecture
3.2.1. Gunshot Detector Device
Hardware
Firmware
3.2.2. Central Server Software
SQL Database
Data Processing in TCP Server
Searching Gunshot Events
3.2.3. Smartphone App
4. Results
4.1. Gunshot Detection Deep Learning Model Results
4.2. Prototype System Results
5. Discussion and Future Work
6. Conclusions
Funding
Data Availability Statement
Conflicts of Interest
References
- Gun Violence Archive. Available online: https://www.gunviolencearchive.org/ (accessed on 22 September 2023).
- Lankford, A. Public Mass Shooters and Firearms: A Cross-National Study of 171 Countries. Violence Vict. 2016, 31, 187–199. [Google Scholar] [CrossRef] [PubMed]
- Number of Mass Shootings in the United States between 1982 and April 2023. Available online: https://www.statista.com/statistics/811487/number-of-mass-shootings-in-the-us/ (accessed on 27 July 2023).
- 49 Killed in Mass Shooting at Two Mosques in Christchurch, New Zealand. Available online: https://www.cnn.com/2019/03/14/asia/christchurch-mosque-shooting-intl/index.html (accessed on 27 July 2023).
- How Frequently Do Church Shootings Occur? Available online: https://lifewayresearch.com/2020/02/12/how-frequently-do-church-shootings-occur/ (accessed on 27 July 2023).
- Effects of Gun Violence. Available online: https://www.bradyunited.org/issue/effects-of-gun-violence (accessed on 27 July 2023).
- SoundThinking. Available online: https://www.soundthinking.com/law-enforcement/gunshot-detection-technology (accessed on 31 July 2023).
- Samireddy, S.R.; Carletta, J.; Lee, K. An embeddable algorithm for gunshot detection. In Proceedings of the 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS), Boston, MA, USA, 6–9 August 2017; pp. 68–71. [Google Scholar] [CrossRef]
- Thanhikam, W. Gunshot noise detection using zero phase technique. In Proceedings of the 2015 Asian Conference on Defence Technology (ACDT), Hua Hin, Thailand, 23–25 April 2015; pp. 183–186. [Google Scholar] [CrossRef]
- MHrabina; Sigmund, M. Gunshot recognition using low level features in the time domain. In Proceedings of the 2018 28th International Conference Radioelektronika (RADIOELEKTRONIKA), Prague, Czech Republic, 19–20 April 2018; pp. 1–5. [Google Scholar] [CrossRef]
- Lopez-Morillas, J.; Canadas-Quesada, F.J.; Vera-Candeas, P.; Ruiz-Reyes, N.; Mata-Campos, R.; Montiel-Zafra, V. Gunshot detection and localization based on Non-negative Matrix Factorization and SRP-Phat. In Proceedings of the 2016 IEEE Sensor Array and Multichannel Signal Processing Workshop (SAM), Rio de Janeiro, Brazil, 10–13 July 2016; pp. 1–5. [Google Scholar] [CrossRef]
- Valenzise, G.; Gerosa, L.; Tagliasacchi, M.; Antonacci, F.; Sarti, A. Scream and gunshot detection and localization for audio-surveillance systems. In Proceedings of the 2007 IEEE Conference on Advanced Video and Signal Based Surveillance, London, UK, 5–7 September 2007; pp. 21–26. [Google Scholar] [CrossRef]
- Chen, C.; Abdallah, A.; Wolf, W. Audiovisual Gunshot Event Recognition. In Proceedings of the 2006 IEEE International Conference on Systems, Man and Cybernetics, Taipei, Taiwan, 8–11 October 2006; pp. 4807–4812. [Google Scholar] [CrossRef]
- Galangque, C.M.J.; Guirnaldo, S.A. Gunshot Classification and Localization System using Artificial Neural Network (ANN). In Proceedings of the 2019 12th International Conference on Information & Communication Technology and System (ICTS), Surabaya, Indonesia, 18 July 2019; pp. 1–5. [Google Scholar] [CrossRef]
- Bajzik, J.; Prinosil, J.; Koniar, D. Gunshot Detection Using Convolutional Neural Networks. In Proceedings of the 2020 24th International Conference Electronics, Palanga, Lithuania, 15–17 June 2020; pp. 1–5. [Google Scholar] [CrossRef]
- Morehead, A.; Ogden, L.; Magee, G.; Hosler, R.; White, B.; Mohler, G. Low Cost Gunshot Detection using Deep Learning on the Raspberry Pi. In Proceedings of the 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA, 9–12 December 2019; pp. 3038–3044. [Google Scholar] [CrossRef]
- Park, J.; Cho, Y.; Sim, G.; Lee, H.; Choo, J. Enemy Spotted: In-game Gun Sound Dataset for Gunshot Classification and Localization. In Proceedings of the 2022 IEEE Conference on Games (CoG), Beijing, China, 21–24 August 2022; pp. 56–63. [Google Scholar] [CrossRef]
- BGG Dataset (PUBG Gun Sound Dataset). Available online: https://github.com/junwoopark92/BG-Gun-Sound-Dataset (accessed on 1 August 2023).
- Jaszczak, B.; Nelson, B. Free Firearm Sound Effects Library. Available online: https://opengameart.org/content/the-free-firearm-sound-library (accessed on 1 August 2023).
- Tuncer, T.; Dogan, S.; Akbal, E.; Aydemir, E. An automated gunshot audio classification method based on finger pattern feature generator and iterative relieff feature selector. ADYU Mühendislik Bilim. Derg. 2021, 8, 225–243. [Google Scholar]
- Gunshot Audio Dataset. Available online: https://www.kaggle.com/datasets/emrahaydemr/gunshot-audio-dataset (accessed on 1 August 2023).
- Gunshot Audio Forensics Dataset. Available online: http://cadreforensics.com/audio/ (accessed on 1 August 2023).
- Kabealo, R.; Wyatt, S.; Aravamudan, A.; Zhang, X.; Acaron, D.N.; Dao, M.P.; Elliott, D.; Smith, A.O.; Otero, C.E.; Otero, L.D.; et al. A multi-firearm, multi-orientation audio dataset of gunshots. Data Brief 2023, 48, 109091. [Google Scholar] [CrossRef] [PubMed]
- Gunshot/Gunfire Audio Dataset. Available online: https://zenodo.org/record/7004819#.Y8WJfHbMK3A (accessed on 1 August 2023).
- Salamon, J.; Jacoby, C.; Bello, J.P. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM International Conference on Multimedia, New York, NY, USA, 3–7 November 2014; pp. 1041–1044. [Google Scholar]
- Urbansound8k Dataset. Available online: https://urbansounddataset.weebly.com/urbansound8k.html (accessed on 1 August 2023).
- Piczak, K.J. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Annual ACM Conference on Multimedia, Brisbane, Australia, 26–30 October 2015; pp. 1015–1018. [Google Scholar]
- ESC-50: Dataset for Environmental Sound Classification. Available online: https://github.com/karolpiczak/ESC-50 (accessed on 1 August 2023).
- Fonseca, E.; Favory, X.; Pons, J.; Font, F.; Serra, X. FSD50K: An Open Dataset of Human-Labeled Sound Events. IEEE/ACM Trans. Audio Speech Lang. Process. 2022, 30, 829–852. [Google Scholar] [CrossRef]
- FSD50K Dataset. Available online: https://zenodo.org/record/4060432 (accessed on 1 August 2023).
- Khan, T. A deep learning model for snoring detection and vibration notification using a smart wearable gadget. Electronics 2019, 8, 987. [Google Scholar] [CrossRef]
- Snoring Dataset. Available online: https://www.kaggle.com/datasets/tareqkhanemu/snoring (accessed on 1 August 2023).
- WavePad Audio Editing Software. Available online: https://www.nch.com.au/wavepad/index.html (accessed on 1 August 2023).
- Pydiogment Library. Available online: https://github.com/SuperKogito/pydiogment/ (accessed on 1 August 2023).
- Pydub Library. Available online: https://github.com/jiaaro/pydub (accessed on 1 August 2023).
- Davis, S.; Mermelstein, P. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Trans. Acoust. Speech Signal Process. 1980, 28, 357–366. [Google Scholar] [CrossRef]
- Fayek, H. Speech Processing for Machine Learning: Filter banks, Mel-Frequency Cepstral Coefficients (MFCCs) and What’s in-between. Available online: https://haythamfayek.com/2016/04/21/speech-processing-for-machine-learning.html (accessed on 2 August 2023).
- SpeechPy. Available online: https://speechpy.readthedocs.io/en/latest/intro/introductions.html (accessed on 2 August 2023).
- LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature 2015, 521, 436–444. [Google Scholar] [CrossRef] [PubMed]
- Nair, V.; Hinton, G.E. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), Haifa, Israel, 21–24 June 2010; pp. 807–814. [Google Scholar]
- Nagi, J.; Ducatelle, F.; Di Caro, G.A.; Cireçsan, D.; Meier, U.; Giusti, A.; Nagi, F.; Schmidhuber, J.; Gambardella, L.M. Max-Pooling Convolutional Neural Networks for Vision-based Hand Gesture Recognition. In Proceedings of the IEEE International Conference on Signal and Image Processing Applications (ICSIPA2011), Kuala Lumpur, Malaysia, 16–18 November 2011. [Google Scholar]
- Glorot, X.; Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, Sardinia, Italy, 13–15 May 2010; pp. 249–256. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 11–18 December 2015; pp. 1026–1034. [Google Scholar]
- Bishop, C.M. Pattern Recognition and Machine Learning; Springer: New York, NY, USA, 2006. [Google Scholar]
- A Look at Gradient Descent and RMSprop Optimizers. Available online: https://towardsdatascience.com/a-look-at-gradient-descent-and-rmsprop-optimizers-f77d483ef08b (accessed on 2 August 2023).
- Keras: The Python Deep Learning Library. Available online: https://keras.io (accessed on 9 August 2023).
- NVIDIA-TensorRT. Available online: https://developer.nvidia.com/tensorrt (accessed on 4 August 2023).
- Accelerating Inference in TensorFlow with TensorRT User Guide. Available online: https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#worflow-with-savedmodel (accessed on 4 August 2023).
- Jetson Nano Developer Kit. Available online: https://developer.nvidia.com/embedded/jetson-nano-developer-kit (accessed on 3 August 2023).
- USB Lavalier Lapel Microphon. Available online: https://www.amazon.com/Lavalier-Microphone-Cardioid-Condenser-K053/dp/B077VNGVL2 (accessed on 3 August 2023).
- Wireless NIC Module for Jetson Nano. Available online: https://www.amazon.com/Wireless-AC8265-Wireless-Developer-Support-Bluetooth/dp/B07V9B5C6M/ (accessed on 3 August 2023).
- Bluetooth Device Configure. Available online: https://manpages.ubuntu.com/manpages/trusty/man8/hciconfig.8.html (accessed on 3 August 2023).
- PyBluez. Available online: https://pybluez.readthedocs.io/en/latest/ (accessed on 3 August 2023).
- Wi-Fi Wrapper Library. Available online: https://pypi.org/project/wifi-wrapper/ (accessed on 3 August 2023).
- Play and Record Sound with Python. Available online: https://python-sounddevice.readthedocs.io/en/0.4.6/ (accessed on 4 August 2023).
- Python Module for Reading and Writing WAV Files. Available online: https://pypi.org/project/wavio/ (accessed on 4 August 2023).
- Date and Time Library. Available online: https://docs.python.org/3/library/datetime.html (accessed on 4 August 2023).
- Socket—Low-Level Networking Interface. Available online: https://docs.python.org/3/library/socket.html (accessed on 3 August 2023).
- HTTP Server. Available online: https://docs.python.org/3/library/http.server.html (accessed on 7 August 2023).
- SQL Server 2022 Express. Available online: https://www.microsoft.com/en-us/sql-server/sql-server-downloads (accessed on 7 August 2023).
- Android Identifiers. Available online: https://developer.android.com/training/articles/user-data-ids (accessed on 8 August 2023).
- FCM Registration Token. Available online: https://firebase.google.com/docs/cloud-messaging/manage-tokens#ensuring-registration-token-freshness (accessed on 8 August 2023).
- C# TCP Server. Available online: https://www.codeproject.com/articles/488668/csharp-tcp-server (accessed on 8 August 2023).
- How to Port Forward. Available online: https://www.noip.com/support/knowledgebase/general-port-forwarding-guide/ (accessed on 8 August 2023).
- How Do I Open a Port on Windows Firewall? Available online: https://www.howtogeek.com/394735/how-do-i-open-a-port-on-windows-firewall/ (accessed on 8 August 2023).
- Thompson, B. C# Database Connection: How to Connect SQL Server. Available online: https://www.guru99.com/c-sharp-access-database.html (accessed on 8 August 2023).
- GMap.NET—Maps for Windows. Available online: https://github.com/judero01col/GMap.NET (accessed on 8 August 2023).
- FcmSharp. Available online: https://github.com/bytefish/FcmSharp (accessed on 8 August 2023).
- GeoLocation. Available online: https://www.b4x.com/android/forum/threads/geolocation.99710/#content (accessed on 8 August 2023).
- Firebase Cloud Messaging. Available online: https://firebase.google.com/docs/cloud-messaging (accessed on 8 August 2023).
- FirebaseNotifications—Push Messages/Firebase Cloud Messaging (FCM). Available online: https://www.b4x.com/android/forum/threads/b4x-firebase-push-notifications-2023.148715/ (accessed on 8 August 2023).
- GeeekPi Nano Case. Available online: https://www.amazon.com/GeeekPi-Support-Developer-Powerful-Computer/dp/B098J4JMLG/ (accessed on 10 August 2023).
- Jetson-Stats. Available online: https://rnext.it/jetson_stats/ (accessed on 10 August 2023).
- Kerampran, C.; Gajewski, T.; Sielicki, P.W. Temperature Measurement of a Bullet in Flight. Sensors 2020, 20, 7016. [Google Scholar] [CrossRef] [PubMed]
- Khan, M.U.; Misbah, M.; Kaleem, Z.; Deng, Y.; Jamalipour, A. GAANet: Ghost Auto Anchor Network for Detecting Varying Size Drones in Dark. In Proceedings of the 2023 IEEE 97th Vehicular Technology Conference (VTC2023-Spring), Florence, Italy, 20–23 June 2023; pp. 1–5. [Google Scholar] [CrossRef]
- Anwar, M.Z.; Kaleem, Z.; Jamalipour, A. Machine Learning Inspired Sound-Based Amateur Drone Detection for Public Safety Applications. IEEE Trans. Veh. Technol. 2019, 68, 2526–2534. [Google Scholar] [CrossRef]
M. Hrabina, et al. [10] | J. Morillas et al. [11] | G. Valenzise et al. [12] | C. Chen et al. [13] | C. Galangque et al. [14] | J. Bajzik et al. [15] | A. Morehead et al. [16] | Proposed | |
---|---|---|---|---|---|---|---|---|
Classifier | ANN | NMF | GMM | SVM | ANN | CNN | CNN | CNN |
Dataset size | 11,004 | 215 | - | 459 videos | 917 | 7000 | <90,000 | 670,000 |
Accuracy | - | - | - | - | 99% | 99% | 99% | 98% |
Precision | 69.3% | - | 93% | 73.46% | - | - | - | 98% |
Embedded system implementation | no | no | no | no | no | no | yes | yes |
Record gunshot with timestamp | no | no | no | no | yes | no | no | yes |
Plot on map | no | no | no | no | no | no | no | yes |
User and device configuration | no | no | no | no | no | no | no | yes |
Database implementation | no | no | no | no | no | no | no | yes |
Smartphone notification | no | no | no | no | no | no | yes (using SMS) | yes |
Training | Validation | Test | |
---|---|---|---|
Loss | 0.0504 | 0.0500 | 0.0505 |
Accuracy | 0.9831 | 0.9832 | 0.9828 |
Precision | Recall | f1-Score | |
---|---|---|---|
Gunshot | 0.98 | 0.99 | 0.98 |
Other | 0.99 | 0.98 | 0.98 |
Hardware Part | Power |
---|---|
Jetson Nano’s CPU | 854 mW |
Jetson Nano’s GPU | 40 mW |
Entire Device | 2.7 W |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Khan, T. Towards an Indoor Gunshot Detection and Notification System Using Deep Learning. Appl. Syst. Innov. 2023, 6, 94. https://doi.org/10.3390/asi6050094
Khan T. Towards an Indoor Gunshot Detection and Notification System Using Deep Learning. Applied System Innovation. 2023; 6(5):94. https://doi.org/10.3390/asi6050094
Chicago/Turabian StyleKhan, Tareq. 2023. "Towards an Indoor Gunshot Detection and Notification System Using Deep Learning" Applied System Innovation 6, no. 5: 94. https://doi.org/10.3390/asi6050094