DeepSDC: Deep Ensemble Learner for the Classification of Social-Media Flooding Events
Abstract
:1. Introduction
- A two-stage ensemble approach is proposed which combines the information obtained by visual and textual data to predict the flooding event. The proposed design provides improved generalization ability with the help of a diverse range of pre-trained networks.
- Combined use of visual and textual information is proposed, where the text data from tweets and corresponding images are used for the flood-detection process.
- The experiments are conducted on three publically available benchmark datasets, including Flood-related Multimedia Task (FRMT) [13], Flood Classification for Social Multimedia (FCSM) [14], and Disaster Image Retrieval from Social Media (DIRSM) [15]. The performance of the proposed DeepSDC is compared to several state-of-the-art flood-detection algorithms.
2. Literature Review
2.1. Social-Media-Based Flood Detection and Response Systems
2.1.1. Studies Related to the FRMT Dataset
2.1.2. Studies Related to the FCSM Dataset
2.1.3. Studies Related to the DIRSM Dataset
3. Methodology
3.1. DeepSDC Module-I: Textual Module
3.1.1. Data Translation and Pre-Processing
3.1.2. Class Balancing
3.1.3. Language Model Combination for Textual Data
3.2. DeepSDC Module-II: Visual Module
3.3. DeepSDC: Ensemble of Module I and Module II
4. Experimental Setup
4.1. Datasets
4.2. Experimental Environment and Details of the Model Training
4.3. Evaluation Measures
4.3.1. Mean Average Precision and Average Precision at K
4.3.2. F1 Score
5. Results and Discussion
5.1. Results of DeepSDC: FRMT Dataset
5.2. Results of DeepSDC: DIRSM Dataset
5.3. Results of DeepSDC: The FCSM Dataset
5.4. Analysis of Why DeepSDC Works
5.5. Analysis of DeepSDC Predictions and Limitations
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- EM-DAT. The International Disaster Database. Center for Research on the Epidemiology of Disasters. 2019. Available online: https://www.emdat.be/ (accessed on 26 February 2023).
- Lopez-Fuentes, L.; Farasin, A.; Zaffaroni, M.; Skinnemoen, H.; Garza, P. Deep Learning Models for Road Passability Detection during Flood Events Using Social Media Data. Appl. Sci. 2020, 10, 8783. [Google Scholar] [CrossRef]
- EU-Commission. Funding Opportunities to Support Disaster Risk Prevention in the Cohesion Policy 2014–2020 Period; European Commission: Brussels, Belgium, 2014. [Google Scholar]
- Esposito, M.; Palma, L.; Belli, A.; Sabbatini, L.; Pierleoni, P. Recent advances in internet of things solutions for early warning systems: A review. Sensors 2022, 22, 2124. [Google Scholar] [CrossRef]
- Wu, R.S.; Sin, Y.Y.; Wang, J.X.; Lin, Y.W.; Wu, H.C.; Sukmara, R.B.; Indawati, L.; Hussain, F. Real-time flood warning system application. Water 2022, 14, 1866. [Google Scholar] [CrossRef]
- Cao, C.; Xu, P.; Wang, Y.; Chen, J.; Zheng, L.; Niu, C. Flash flood hazard susceptibility mapping using frequency ratio and statistical index methods in coalmine subsidence areas. Sustainability 2016, 8, 948. [Google Scholar] [CrossRef] [Green Version]
- Khan, M.Y.A.; ElKashouty, M.; Subyani, A.M.; Tian, F. Flash Flood Assessment and Management for Sustainable Development Using Geospatial Technology and WMS Models in Abha City, Aseer Region, Saudi Arabia. Sustainability 2022, 14, 10430. [Google Scholar] [CrossRef]
- Sarkar, S.K.; Ansar, S.B.; Ekram, K.M.M.; Khan, M.H.; Talukdar, S.; Naikoo, M.W.; Islam, A.R.T.; Rahman, A.; Mosavi, A. Developing robust flood susceptibility model with small numbers of parameters in highly fertile regions of Northwest Bangladesh for sustainable flood and agriculture management. Sustainability 2022, 14, 3982. [Google Scholar] [CrossRef]
- Pandey, A.C.; Kaushik, K.; Parida, B.R. Google Earth Engine for large-scale flood mapping using SAR data and impact assessment on agriculture and population of Ganga-Brahmaputra basin. Sustainability 2022, 14, 4210. [Google Scholar] [CrossRef]
- Attia, W.; Ragab, D.; Abdel-Hamid, A.M.; Marghani, A.M.; Elfadaly, A.; Lasaponara, R. On the Use of Radar and Optical Satellite Imagery for the Monitoring of Flood Hazards on Heritage Sites in Southern Sinai, Egypt. Sustainability 2022, 14, 5500. [Google Scholar] [CrossRef]
- Werner, M.; Reggiani, P.; De Roo, A.; Bates, P.; Sprokkereef, E. Flood forecasting and warning at the river basin and at the European scale. Nat. Hazards 2005, 36, 25–42. [Google Scholar] [CrossRef]
- George, E.I.; Abraham, C.M. Real-time earthquake detection using Twitter tweets. In AIP Conference Proceedings; AIP Publishing LLC.: Melville, NY, USA, 2022; Volume 2520, p. 030014. [Google Scholar]
- Andreadis, S.; Gialampoukidis, I.; Karakostas, A.; Vrochidis, S.; Kompatsiaris, I.; Fiorin, R.; Norbiato, D.; Ferri, M. The flood-related multimedia task at mediaeval 2020. In Proceedings of the MediaEval 2020 Workshop, Online, 14–15 December 2020; pp. 14–15. [Google Scholar]
- Benjamin, B.; Patrick, H.; Zhengyu, Z.; Damian, B. The multimedia satellite task at mediaeval 2018: Emergency response for flooding events. In Proceedings of the MediaEval 2018 Workshop (CEUR Workshop Proceedings), Sophia Antipolis, France, 29–31 October 2018. [Google Scholar]
- Bischke, B.; Helber, P.; Schulze, C.; Venkat, S.; Dengel, A.; Borth, D. The multimedia satellite task at mediaeval 2017: Emergence response for flooding events. In Proceedings of the MediaEval 2017 Workshop, Dublin, Ireland, 13–15 September 2017. [Google Scholar]
- Said, N.; Ahmad, K.; Gul, A.; Ahmad, N.; Al-Fuqaha, A. Floods Detection in Twitter Text and Images. In Proceedings of the MediaEval 2020 Workshop, Online, 14–15 December 2020. [Google Scholar]
- Islam, R.; Alan, W. Flood Detection in Twitter Using a Novel Learning Method for Neural Networks. In Proceedings of the MediaEval 2020 Workshop, Online, 14–15 December 2020. [Google Scholar]
- Hanif, M.; Joozer, M.; Huzaifa, M.; Rafi, M. An ensemble based method for the classification of flooding event using social media data. In Proceedings of the MediaEval 2018 Workshop, Sophia Antipolis, France, 29–31 October 2018; pp. 29–31. [Google Scholar]
- Avgerinakis, K.; Moumtzidou, A.; Andreadis, S.; Michail, E.; Gialampoukidis, I.; Vrochidis, S.; Kompatsiaris, I. Visual and textual analysis of social media and satellite images for flood detection@ multimedia satellite task MediaEval 2017. In Proceedings of the MediaEval 2017 Workshop, Dublin, Ireland, 13–15 September 2017. [Google Scholar]
- Zhao, Z.; Larson, M.A. Retrieving Social Flooding Images Based on Multimodal Information. In Proceedings of the MediaEval 2017 Workshop, Dublin, Ireland, 13–15 September 2017. [Google Scholar]
- Nogueira, K.; Fadel, S.G.; Dourado, Í.C.; de Oliveira Werneck, R.; Muñoz, J.A.; Penatti, O.A.; Calumby, R.T.; Li, L.; dos Santos, J.A.; da Silva Torres, R. Data-Driven Flood Detection using Neural Networks. In Proceedings of the MediaEval 2017 Workshop, Dublin, Ireland, 13–15 September 2017. [Google Scholar]
- Dao, M.S.; Pham, Q.N.M.; Dang Nguyen, D.T. A domain-based late-fusion for disaster image retrieval from social media. In Proceedings of the MediaEval 2017 Workshop, Dublin, Ireland, 13–15 September 2017. [Google Scholar]
- Ahmad, S.; Ahmad, K.; Ahmad, N.; Conci, N. Convolutional Neural Networks for Disaster Images Retrieval. In Proceedings of the MediaEval 2017 Workshop, Dublin, Ireland, 13–15 September 2017. [Google Scholar]
- Bischke, B.; Bhardwaj, P.; Gautam, A.; Helber, P.; Borth, D.; Dengel, A. Detection of Flooding Events in Social Multimedia and Satellite Imagery using Deep Neural Networks. In Proceedings of the MediaEval 2017 Workshop, Dublin, Ireland, 13–15 September 2017. [Google Scholar]
- Dourado, I.C.; Tabbone, S.; da Silva Torres, R. Event prediction based on unsupervised graph-based rank-fusion models. In Proceedings of the International Workshop on Graph-Based Representations in Pattern Recognition, Tours, France, 19–21 June 2019; Springer: Berlin, Germany, 2019; pp. 88–98. [Google Scholar]
- Hanif, M.; Tahir, M.A.; Rafi, M. VRBagged-Net: Ensemble Based Deep Learning Model for Disaster Event Classification. Electronics 2021, 10, 1411. [Google Scholar] [CrossRef]
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
- Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv 2019, arXiv:1907.11692. [Google Scholar]
- Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R.R.; Le, Q.V. Xlnet: Generalized autoregressive pretraining for language understanding. In Proceedings of the Advances in Neural Information Processing Systems 32, Vancouver, BC, Canada, 8–14 December 2019; Volume 32. [Google Scholar]
- Lopez-Fuentes, L.; Farasin, A.; Skinnemoen, H.; Garza, P. Deep Learning Models for Passability Detection of Flooded Roads. In Proceedings of the MediaEval 2018 Workshop (CEUR Workshop Proceedings), Sophia Antipolis, France, 29–31 October 2018. [Google Scholar]
- Zhao, Z.; Larson, M.; Oostdijk, N. Exploiting Local Semantic Concepts for Flooding-related Social Image Classification. In Proceedings of the MediaEval 2018 Workshop (CEUR Workshop Proceedings), Sophia Antipolis, France, 29–31 October 2018. [Google Scholar]
- Bischke, B.; Helber, P.; Dengel, A. Global-Local Feature Fusion for Image Classification of Flood Affected Roads from Social Multimedia. In Proceedings of the 16th ACM International Conference on Multimedia (CEUR Workshop Proceedings), Vancouver, BC, Canada, 26–31 October 2008; pp. 1085–1088. [Google Scholar]
- Jou, B.; Chang, S.F. Deep cross residual learning for multitask visual recognition. In Proceedings of the 24th ACM International Conference on Multimedia, Amsterdam, The Netherlands, 15–19 October 2016; pp. 998–1007. [Google Scholar]
- Chen, T.; Borth, D.; Darrell, T.; Chang, S.F. Deepsentibank: Visual sentiment concept classification with deep convolutional neural networks. arXiv 2014, arXiv:1410.8586. [Google Scholar]
- Tkachenko, N.; Zubiaga, A.; Procter, R. Wisc at Mediaeval 2017: Multimedia Satellite Task. 2017. Available online: https://qmro.qmul.ac.uk/xmlui/handle/123456789/56411 (accessed on 26 February 2023).
- Sun, Y.; Wong, A.K.; Kamel, M.S. Classification of imbalanced data: A review. Int. J. Pattern Recognit. Artif. Intell. 2009, 23, 687–719. [Google Scholar] [CrossRef]
- Barandela, R.; Sánchez, J.S.; Garcıa, V.; Rangel, E. Strategies for learning in class imbalance problems. Pattern Recognit. 2003, 36, 849–851. [Google Scholar] [CrossRef]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; Volume 30. [Google Scholar]
- Jagadeesh, M.; Alphonse, P. NIT_COVID-19 at WNUT-2020 Task 2: Deep Learning Model RoBERTa for Identify Informative COVID-19 English Tweets. In Proceedings of the W-NUT@ EMNLP, Online, 19 November 2020; pp. 450–454. [Google Scholar]
- Abavisani, M.; Wu, L.; Hu, S.; Tetreault, J.; Jaimes, A. Multimodal categorization of crisis events in social media. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; pp. 14679–14689. [Google Scholar]
- Howard, J. Fastai. 2018. Available online: https://github.com/fastai/fastai (accessed on 26 February 2023).
- Dai, Z.; Yang, Z.; Yang, Y.; Carbonell, J.; Le, Q.V.; Salakhutdinov, R. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv 2019, arXiv:1901.02860. [Google Scholar]
- Bloice, M.; Stocker, C.; Holzinger, A. Augmentor: An Image Augmentation Library for Machine Learning. J. Open Source Softw. 2017, 2, 432. [Google Scholar] [CrossRef]
- Waqas, M.; Tahir, M.A.; Khan, S.A. Robust bag classification approach for multi-instance learning via subspace fuzzy clustering. Expert Syst. Appl. 2023, 214, 119113. [Google Scholar] [CrossRef]
- Waqas, M.; Tahir, M.A.; Qureshi, R. Deep Gaussian mixture model based instance relevance estimation for multiple instance learning applications. Appl. Intell. 2022, 1–16. [Google Scholar] [CrossRef]
- Waqas, M.; Khan, Z.; Anjum, S.; Tahir, M.A. Lung-Wise Tuberculosis Analysis and Automatic CT Report Generation with Hybrid Feature and Ensemble Learning. In Proceedings of the CLEF (Working Notes), Thessaloniki, Greece, 22–25 September 2020. [Google Scholar]
- Waqas, M.; Tahir, M.A.; Qureshi, R. Ensemble-Based Instance Relevance Estimation in Multiple-Instance Learning. In Proceedings of the 2021 9th European Workshop on Visual Information Processing (EUVIP), Paris, France, 23–25 June 2021; pp. 1–6. [Google Scholar]
- Zhang, Z. Improved adam optimizer for deep neural networks. In Proceedings of the 2018 IEEE/ACM 26th International Symposium on Quality of Service (IWQoS), Banff, AB, Canada, 4–6 June 2018; pp. 1–2. [Google Scholar]
- Zhang, Z.; Sabuncu, M. Generalized cross entropy loss for training deep neural networks with noisy labels. In Proceedings of the Advances in Neural Information Processing Systems 31 (NeurIPS 2018), Montreal, QC, Canada, 3–8 December 2018; Volume 31. [Google Scholar]
- Zhou, B.; Lapedriza, A.; Khosla, A.; Oliva, A.; Torralba, A. Places: A 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 40, 1452–1464. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Deng, J.; Dong, W.; Socher, R.; Li, L.J.; Li, K.; Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 248–255. [Google Scholar]
- Nikoletopoulos, T.; Wolff, C. A Tweet Text Binary Artificial Neural Network Classifier. In Proceedings of the MediaEval, Online, 14–15 December 2020. [Google Scholar]
- Chollet, F. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1251–1258. [Google Scholar]
- Lopez-Fuentes, L.; van de Weijer, J.; Bolanos, M.; Skinnemoen, H. Multi-modal Deep Learning Approach for Flood Detection. In Proceedings of the MediaEval 2017 Workshop, Dublin, Ireland, 13–15 September 2017. [Google Scholar]
- Hanif, M.; Tahir, M.A.; Rafi, M. Detection of passable roads using Ensemble of Global and Local. In Proceedings of the MediaEval 2018 Workshop (CEUR Workshop Proceedings), Sophia Antipolis, France, 29–31 October 2018. [Google Scholar]
Dataset | Year | Objective | Nature | Size | Evaluation Measure |
---|---|---|---|---|---|
Disaster Image Retrieval from Social Media (DIRSM) [15]. | 2017 | Flood-event detection | Textual and visual | 6600 | Average Precision at K and Mean Average Precision |
Flood Classification for Social Multimedia (FCSM) [14]. | 2018 | Detection of passable roads | Textual and visual | 8844 | F1 Score |
Flood-related Multimedia Task (FRMT) [13]. | 2020 | Flood-event detection using Italian language tweets | Textual and visual | 7777 | F1 Score |
Methods | Techniques Used | F-Score |
---|---|---|
Hanif et al. [18] | TF-IDF, Multinomial Naive Bayes | 36.31 |
Rabiul et al. [17] | TF-IDF, Multinomial Naive Bayes | 41.58 |
Naina et al. [16] | Bag of Words | 43.70 |
Nikoletopoulos et al. [53] | ANN, Undersampling | 54.05 |
DeepSDC (Module-I) | Ensemble of Roberta and XLNet | 57.92 |
Methods | Techniques Used | F-Score |
---|---|---|
Naina et al. [16] | BoW, Resnet, VGG | 09.00 |
Rabiul et al. [17] | TF-IDF, Chi-squared, Xception | 14.78 |
Hanif et al. [18] | TF-IDF, Multinomial Nave Bayes, VGG16 | 27.86 |
DeepSDC (Module-I & II) | Ensemble of Visual and Textual Module | 46.52 |
Methods | Techniques Used | AP@480 | MAP |
---|---|---|---|
Dao et al. [22] | BoW | 57.07 | 57.12 |
Keiller et al. [21] | BoW, Relation Network | 76.71 | 62.63 |
Laura et al. [55] | Glove, LSTM | 61.58 | 66.38 |
Nataliya et al. [36] | BoW, Logistic Regression | 66.78 | 74.37 |
Zhengyu et al. [20] | Ranking based fusion | 63.7 | 75.74 |
DeepSDC (Module-I) | Ensemble of XLNet and RoBERTa | 76.33 | 91.29 |
Methods | Techniques Used | AP@480 | MAP |
---|---|---|---|
Laura et al. [55] | Glove, LSTM, Inception | 83.96 | 81.6 |
Zhengyu et al. [20] | Ranking, CEDD, SVM | 85.43 | 73.16 |
Keiller et al. [21] | Relation Network, ResNet | 85.63 | 95.84 |
Dao et al. [22] | BoW, SVM, CNN | 90.39 | 85.41 |
Sheharyar et al. [23] | AlexNet, SVM | 92.55 | 83.73 |
DeepSDC(M-I & M-II) | VGG, ResNet, Roberta, XLNet | 92.65 | 98.94 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Hanif, M.; Waqas, M.; Muneer, A.; Alwadain, A.; Tahir, M.A.; Rafi, M. DeepSDC: Deep Ensemble Learner for the Classification of Social-Media Flooding Events. Sustainability 2023, 15, 6049. https://doi.org/10.3390/su15076049
Hanif M, Waqas M, Muneer A, Alwadain A, Tahir MA, Rafi M. DeepSDC: Deep Ensemble Learner for the Classification of Social-Media Flooding Events. Sustainability. 2023; 15(7):6049. https://doi.org/10.3390/su15076049
Chicago/Turabian StyleHanif, Muhammad, Muhammad Waqas, Amgad Muneer, Ayed Alwadain, Muhammad Atif Tahir, and Muhammad Rafi. 2023. "DeepSDC: Deep Ensemble Learner for the Classification of Social-Media Flooding Events" Sustainability 15, no. 7: 6049. https://doi.org/10.3390/su15076049