Two-Step Real-Time Night-Time Fire Detection in an Urban Environment Using Static ELASTIC-YOLOv3 and Temporal Fire-Tube
Abstract
:1. Introduction
- a loss of color information;
- a relatively high brightness value compared to the surroundings;
- various changes in the shape and size from light blurring; and
- movements of the flames in all directions (daytime flames tend to move in an upward direction).
2. Related Studies
2.1. Machine Learning and Deep Learning-Based Fire Detection
2.2. Contributions of this Work
3. Fire Detection Methods Using ELASTIC-YOLOv3 and Fire-Tube Analysis
3.1. ELASTIC-YOLOv3
3.2. Temporal Fire-Tube Generation Using the Histogram of Optical Flow
Algorithm 1 Skip Frame for Fire-Tube Generation |
: A set of fire-tubes |
Initialize , N: size of F-Tube |
: Magnitude of HoF |
th: a threshold for skip frame selection |
k: index of fire-tube, i: index of current candidate frame (i > N (50)) |
K = 0, |
i-- |
Do |
Then |
F-Tube [k] = frame (i) |
k++, i-- |
Else |
i--//skip frame |
While (No. of frames of F-Tube is full) |
3.3. Bag-of-Features Extraction from HoF and Fire Verification
4. Experimental Results
4.1. Performance Evaluation of Fire Candidate Detection
4.2. Performance Evaluation of Fire Candidate Detection
4.3. Performance Evaluation of Temporal Fire-Tube and BoF
5. Conclusions
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
References
- Vescoukis, V.; Doulamis, N.; Karagiorgou, S. A service oriented architecture for decision support systems in environmental crisis management. Future Gener. Comput. Syst. 2012, 28, 593–604. [Google Scholar] [CrossRef]
- Ko, B.C.; Kwak, S.Y. A survey of computer vision-based natural disaster warning systems. Opt. Eng. 2012, 51, 070901. [Google Scholar] [CrossRef] [Green Version]
- Korea National Fire Agency. Fire Occurrence Status. Available online: http://www.index.go.kr/potal/main/EachDtlPageDetail.do?idx_cd=1632 (accessed on 3 March 2020).
- Ko, B.C.; Ham, S.J.; Nam, J.Y. Modeling and formalization of fuzzy finite automata for detection of irregular fire flames. IEEE Trans. Circuits Syst. Video Technol. 2011, 21, 1903–1912. [Google Scholar] [CrossRef]
- Ko, B.C.; Jung, J.H.; Nam, J.Y. Fire detection and 3D surface reconstruction based on stereoscopic pictures and probabilistic fuzzy logic. Fire Saf. J. 2014, 68, 61–70. [Google Scholar] [CrossRef]
- Chacon, M.; Perez-Vargas, F.J. Thermal video analysis for fire detection using shape regularity and intensity saturation features. In Proceedings of the Third Mexican Conference on Pattern Recognition (MCPR), Cancun, Mexico, 29 June–2 July 2011; pp. 118–126. [Google Scholar]
- Jeong, M.; Ko, B.C.; Nam, J.Y. Early detection of sudden pedestrian crossing for safe driving during summer nights. IEEE Trans. Circuits Syst. Video Technol. 2017, 27, 1368–1380. [Google Scholar] [CrossRef]
- Celik, T.; Demirel, H. Fire detection in video sequences using a generic color model. Fire Saf. J. 2009, 44, 147–158. [Google Scholar] [CrossRef]
- Foggia, P.; Saggese, A.; Vento, M. Real-time fire detection for video-surveillance applications using a combination of experts based on color, shape, and motion. IEEE Trans. Circuits Syst. Video Technol. 2015, 25, 1545–1556. [Google Scholar] [CrossRef]
- Celik, T.; Ozkaramanlt, H.; Demirel, H. Fire pixel classification using fuzzy logic and statistical color model. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Honolulu, HI, USA, 15–20 April 2007; pp. 1205–1208. [Google Scholar]
- Ko, B.C.; Cheong, K.H.; Nam, J.Y. Fire detection based on vision sensor and support vector machines. Fire Saf. J. 2009, 44, 322–329. [Google Scholar] [CrossRef]
- Borges, P.V.K.; Izquierdo, E. A probabilistic approach for vision-based fire detection in videos. IEEE Trans. Circuits Syst. Video Technol. 2010, 20, 721–731. [Google Scholar] [CrossRef]
- Ko, B.; Cheong, K.H.; Nam, J.Y. Early fire detection algorithm based on irregular patterns of flames and hierarchical Bayesian Networks. Fire Saf. J. 2010, 45, 262–270. [Google Scholar] [CrossRef]
- Mueller, M.; Karasev, P.; Kolesov, I.; Tannenbaum, A. Optical flow estimation for flame detection in videos. IEEE Trans. Image Process. 2013, 22, 2786–2797. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Kim, O.; Kang, D.J. Fire detection system using random forest classification for image sequences of complex background. Opt. Eng. 2013, 52, 1–10. [Google Scholar] [CrossRef] [Green Version]
- Yann, L.; Bottou, L.; Bengio, Y.; Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 1998, 86, 2278–2324. [Google Scholar]
- Sak, H.; Senior, A.; Beaufays, F. Long Short-term memory recurrent neural network architectures for large scale acoustic modeling. arXiv 2014, arXiv:1402.1128. [Google Scholar]
- Hochreiter, S.; Schmidhuber, J. Long short-term memory. Neural Comput. 1997, 9, 1735–1780. [Google Scholar] [CrossRef] [PubMed]
- Chen, T.H.; Wu, P.H.; Chiou, Y.C. An early fire-detection method based on image processing. In Proceedings of the International Conference on Image Processing (ICIP), Singapore, 24–27 October 2004; pp. 1707–1710. [Google Scholar]
- Töreyin, B.U.; Dedeoğlu, Y.; Güdükbay, U.; Cetin, A.E. Computer vision based method for real-time fire and flame detection. Pattern Recognit. Lett. 2006, 27, 49–58. [Google Scholar] [CrossRef] [Green Version]
- Ko, B.C.; Hwang, H.J.; Nam, J.Y. Nonparametric membership functions and fuzzy logic for vision sensor-based flame detection. Opt. Eng. 2010, 49, 127202. [Google Scholar] [CrossRef]
- Dimitropoulos, K.; Barmpoutis, P.; Nikos, G. Spatio-Temporal flame modeling and dynamic texture analysis for automatic video-based fire detection. IEEE Trans. Circuits Syst. Video Technol. 2015, 25, 339–351. [Google Scholar] [CrossRef]
- Frizzi, S.; Kaabi, R.; Bouchouicha, M.; Ginoux, J.M.; Moreau, E.; Fnaiech, F. Convolutional neural network for video fire and smoke detection. In Proceedings of the IEEE Annual Conference of the IEEE Industrial Electronics Society( IECON), Florence, Italy, 23–26 October 2016; pp. 877–882. [Google Scholar]
- Zhang, Q.; Xu, J.; Xu, L.; Guo, H. Deep convolutional neural networks for forest fire detection. In Proceedings of the International Forum on Management, Education and Information Technology Application (IFMEITA), Guangzhou, China, 30–31 January 2016; pp. 568–575. [Google Scholar]
- Muhammad, K.; Ahmad, J.; Lv, Z.; Bellavista, P.; Yang, P.; Baik, S.W. Efficient deep CNN-based fire detection and localization in video surveillance applications. IEEE Trans. Syst. Man Cybern. Syst. 2018, 4, 1419–1434. [Google Scholar] [CrossRef]
- Iandola, F.N.; Moskewicz, M.W.; Ashraf, K.; Han, S.; Dally, W.J.; Keutzer, K. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1 MB model size. arXiv 2016, arXiv:1602.07360. [Google Scholar]
- Muhammad, K.; Ahmad, J.; Mehmood, I.; Rho, S.; Baik, S.W. Convolutional neural networks based fire detection in surveillance videos. IEEE Access 2018, 6, 18174–18183. [Google Scholar] [CrossRef]
- Szegedy, C.; Liu, W.; Sermanet, P.; Reed, S.; Anguelow, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015; pp. 1–9. [Google Scholar]
- Dunnings, A.J.; Breckon, T.P. Experimentally defined convolutional neural network architecture variants for non-temporal real-time fire detection. In Proceedings of the IEEE International Conference on Image Processing (ICIP), Athens, Greece, 7–10 October 2018; pp. 1–5. [Google Scholar]
- Barmpoutis, P.; Dimitropoulos, K.; Kaza, K.; Grammalidis, N. Fire detection from images using faster R-CNN and multidimensional texture analysis. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 12–17 May 2019; pp. 8301–8305. [Google Scholar]
- Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 6, 1137–1149. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Han, J.S.; Kim, G.S.; Han, Y.K.; Lee, C.S.; Kim, S.H.; Hwang, U. Predictive models of fire via deep learning exploiting colorific variation. In Proceedings of the International Conference on Artificial Intelligence in Information and Communication (ICAIIC), Okinawa, Japan, 11–13 February 2019; pp. 579–581. [Google Scholar]
- Cao, Y.; Yang, F.; Tang, Q.; Lu, X. An attention enhanced bidirectional LSTM for early forest fire smoke recognition. IEEE Access 2019, 7, 154732–154742. [Google Scholar] [CrossRef]
- Kim, B.; Lee, J. A video-based fire detection using deep learning models. Appl. Sci. 2019, 9, 2862. [Google Scholar] [CrossRef] [Green Version]
- Park, M.J.; Son, D.; Ko, B.C. Two-Step cascading algorithm for camera-based night fire detection. In Proceedings of the IS&T International Symposium on Electronic Imaging 2020 (EI), Burlingame, CA, USA, 26–30 January 2020; pp. 1–4. [Google Scholar]
- Redmon, J.; Farhadi, A. YOLOv3: An incremental improvement. arXiv 2018, arXiv:1804.02767. [Google Scholar]
- Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 1–10. [Google Scholar]
- Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.Y.; Berg, A.C. SSD: Single shot multibox detector. In Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands, 8–16 October 2016; pp. 1–17. [Google Scholar]
- Wang, H.; Kembhavi, A.; Farhadi, A.; Yuille, A.; Rastegari, M. ELASTIC: Improving CNNs with dynamic scaling policies. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 16–20 June 2019; pp. 2258–2267. [Google Scholar]
- Ko, B.C.; Park, J.O.; Nam, J.Y. Spatiotemporal bag-of-features for early wildfire smoke detection. Image Vis. Comput. 2013, 31, 786–795. [Google Scholar] [CrossRef]
- Lazebnik, S.; Schmid, C.; Ponce, J. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New York, NY, USA, 17–22 June 2006; pp. 1–8. [Google Scholar]
- Kim, S.; Ko, B.C. Building deep random ferns without backpropagation. IEEE Access 2020, 8, 8533–8542. [Google Scholar] [CrossRef]
- Arık, S.O.; Pfister, T. TabNet: Attentive interpretable tabular learning. arXiv 2020, arXiv:1908.07442v4. [Google Scholar]
- Ko, B.C.; Kim, S.H.; Nam, J.Y. X-ray image classification using random forests with local wavelet-based CS-local binary patterns. J. Digit. Imaging 2011, 24, 1141–1151. [Google Scholar] [CrossRef] [Green Version]
- VisiFire. Available online: http://signal.ee.bilkent.edu.tr/VisiFire/ (accessed on 3 March 2020).
- KMU-Fire. Available online: https://cvpr.kmu.ac.kr/Dataset/Dataset.htm (accessed on 3 March 2020).
- Wu, S.; Zhang, L. Using popular object detection methods for real time forest fire detection. In Proceedings of the 11th International Symposium on Computational Intelligence and Design (SCID), Hangzhou, China, 8–9 December 2018; pp. 280–284. [Google Scholar]
- Tran, D.; Bourdev, L.; Fergus, R.; Torresani, L.; Paluri, M. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the International Conference on Computer Vision (ICCV), Santiago, Chile, 11–18 December 2015; pp. 4489–4497. [Google Scholar]
Fire Video No. | Total No. of Frames | Non-Fire No. | Total No. of Frames |
---|---|---|---|
real-fire01 | 1788 | non-fire01 | 343 |
real-fire02 | 862 | non-fire02 | 1439 |
real-fire03 | 1187 | non-fire03 | 1388 |
real-fire04 | 1131 | non-fire04 | 913 |
real-fire05 | 683 | non-fire05 | 575 |
real-fire06 | 1121 | non-fire06 | 514 |
real-fire07 | 1787 | non-fire07 | 1269 |
real-fire08 | 1786 | non-fire08 | 686 |
real-fire09 | 1260 | non-fire09 | 665 |
real-fire10 | 718 | non-fire10 | 473 |
Average | 1232 | Average | 827 |
Methods | Precision (%) | Recall (%) | F1-Score (%) | |
---|---|---|---|---|
YOLOv3 [36] | Small | 97.5 | 81.9 | 89.0 |
Medium | 98.8 | 92.6 | 95.6 | |
Large | 97.6 | 99.2 | 98.4 | |
Average | 98.4 | 90.3 | 94.2 | |
Proposed ELASTIC-YOLOv3 | Small | 96.3 | 92.1 | 94.1 |
Medium | 99.8 | 98.7 | 99.3 | |
Large | 99.4 | 100 | 99.7 | |
Average | 98.8 | 97.0 | 97.9 |
Methods | Precision (%) | Recall (%) | F1-Score (%) | Processing Time (ms) |
---|---|---|---|---|
SSD [38] | 99.0 | 67.7 | 80.4 | 22.5 |
Faster R-CNN [47] | 81.7 | 94.5 | 87.2 | 45 |
YOLOv3 [36] | 97.9 | 91.2 | 94.3 | 15.9 |
ELASTIC-YOLOv3 | 98.5 | 96.9 | 97.7 | 16 |
Methods | Precision (%) | Recall (%) | F1-Score (%) | Processing Time (ms) | Operation |
---|---|---|---|---|---|
ELASTIC-YOLOv3 + 3D ConvNets [48] | 97.9 | 89.3 | 89.3 | 20.4 | GPU |
ELASTIC-YOLOv3 + LSTM [33] | 76.7 | 99.5 | 86.6 | 56 | GPU |
ELASTIC-YOLOv3 + RF classifier | 96.7 | 99.5 | 98.0 | 24.8 | GPU (YOLOv3) + CPU (RF) |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Park, M.; Ko, B.C. Two-Step Real-Time Night-Time Fire Detection in an Urban Environment Using Static ELASTIC-YOLOv3 and Temporal Fire-Tube. Sensors 2020, 20, 2202. https://doi.org/10.3390/s20082202
Park M, Ko BC. Two-Step Real-Time Night-Time Fire Detection in an Urban Environment Using Static ELASTIC-YOLOv3 and Temporal Fire-Tube. Sensors. 2020; 20(8):2202. https://doi.org/10.3390/s20082202
Chicago/Turabian StylePark, MinJi, and Byoung Chul Ko. 2020. "Two-Step Real-Time Night-Time Fire Detection in an Urban Environment Using Static ELASTIC-YOLOv3 and Temporal Fire-Tube" Sensors 20, no. 8: 2202. https://doi.org/10.3390/s20082202