Classification of Rainfall Intensity and Cloud Type from Dash Cam Images Using Feature Removal by Masking
Abstract
:1. Introduction
1.1. Weather Report
1.2. Related Work
1.2.1. Weather Monitoring by Image Recognition
1.2.2. Feature Areas in Weather Recognition Using Images
2. Methods
2.1. Proposal Automatic Weather Reporting System
2.1.1. Step 1 Automatic Image Acquisition
2.1.2. Step 2 Selection of Sky Images
2.1.3. Step 3 Weather Recognition
2.1.4. Step 4 Rainfall Intensity Level Classification
2.1.5. Step 5 Cloud Type Classification
2.1.6. Step 6 Automatic Report
2.2. Feature Removal by Masking
3. Experiments
3.1. Rainfall Intensity Level Classification
3.1.1. Dataset
3.1.2. Model
3.1.3. Results and Discussion
3.2. Cloud Type Classification
3.2.1. Dataset
3.2.2. Model
3.2.3. Results and Discussion
4. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Agency, J.M. Verification of Precipitation. Available online: https://www.data.jma.go.jp/fcd/yoho/kensho/explanation.html (accessed on 21 April 2024).
- Agency, J.M. Weather Warnings and Advisories and Weather Forecast Areas. Available online: https://www.jma.go.jp/jma/kishou/know/saibun/index.html (accessed on 21 April 2024).
- Weathernews, Inc. Accuracy Remains High through Winter 2022 Weather News Weather Forecast Accuracy. Available online: https://weathernews.jp/s/topics/202212/230215/ (accessed on 6 January 2023).
- Aniraj, A.; Dantas, C.F.; Ienco, D.; Marcos, D. Masking Strategies for Background Bias Removal in Computer Vision Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France, 2–6 October 2023; pp. 4397–4405. [Google Scholar]
- Sun, Z.; Wang, P.; Jin, Y.; Wang, J.; Wang, L. A practical weather detection method built in the surveillance system currently used to monitor the large-scale freeway in China. IEEE Access 2020, 8, 112357–112367. [Google Scholar] [CrossRef]
- Tan, L.; Xuan, D.; Xia, J.; Wang, C. Weather recognition based on 3C-CNN. KSII Trans. Internet Inf. Syst. (TIIS) 2020, 14, 3567–3582. [Google Scholar]
- Lu, C.; Lin, D.; Jia, J.; Tang, C.K. Two-class weather classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 3718–3725. [Google Scholar]
- Ye, L.; Cao, Z.; Xiao, Y. DeepCloud: Ground-based cloud image categorization using deep convolutional features. IEEE Trans. Geosci. Remote Sens. 2017, 55, 5729–5740. [Google Scholar] [CrossRef]
- Zhang, J.; Liu, P.; Zhang, F.; Song, Q. CloudNet: Ground-Based Cloud Classification With Deep Convolutional Neural Network. Geophys. Res. Lett. 2018, 45, 8665–8672. [Google Scholar] [CrossRef]
- Lin, P.W.; Hsu, C.M. Lightweight Convolutional Neural Networks with Model-Switching Architecture for Multi-Scenario Road Semantic Segmentation. Appl. Sci. 2021, 11, 7424. [Google Scholar] [CrossRef]
- Weathernews, Inc. Industry’s First! 250-Meter Mesh Raincloud Radar with 10-Minute Intervals for up to 15 Hours ahead New Rain Cloud Radar Available on Weather News App. Available online: https://jp.weathernews.com/news/32149/ (accessed on 31 December 2023).
- Suemitsu, K.; Endo, S.; Sato, S. Selection of Dash Cam Images for Weather Forecasting Based on The Sky Occupancy. In Proceedings of the 2022 Joint 12th International Conference on Soft Computing and Intelligent Systems and 23rd International Symposium on Advanced Intelligent Systems (SCIS&ISIS), Ise, Japan, 29 November–2 December 2022; IEEE: Piscataway, NJ, USA, 2022; pp. 1–8. [Google Scholar]
- Chen, L.C.; Zhu, Y.; Papandreou, G.; Schroff, F.; Adam, H. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 801–818. [Google Scholar]
- Bartos, M.; Park, H.; Zhou, T.; Kerkez, B.; Vasudevan, R. Windshield wipers on connected vehicles produce high-accuracy rainfall maps. Sci. Rep. 2019, 9, 170. [Google Scholar] [CrossRef] [PubMed]
- Yu, F.; Chen, H.; Wang, X.; Xian, W.; Chen, Y.; Liu, F.; Madhavan, V.; Darrell, T. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; pp. 2636–2645. [Google Scholar]
- Yin, H.; Zheng, F.; Duan, H.F.; Savic, D.; Kapelan, Z. Estimating rainfall intensity using an image-based deep learning model. Engineering 2023, 21, 162–174. [Google Scholar] [CrossRef]
- Zheng, F.; Yin, H.; Ma, Y.; Duan, H.F.; Gupta, H.; Savic, D.; Kapelan, Z. Toward Improved Real-Time Rainfall Intensity Estimation Using Video Surveillance Cameras. Water Resour. Res. 2023, 59, e2023WR034831. [Google Scholar] [CrossRef]
- Agency, J.M. Historical Weather Data Search. Available online: https://www.data.jma.go.jp/obd/stats/etrn/index.php (accessed on 1 September 2022).
- Tan, M.; Le, Q. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA, 9–15 June 2019; pp. 6105–6114. [Google Scholar]
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–36 June 2016; pp. 770–778. [Google Scholar]
- Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv 2020, arXiv:2010.11929. [Google Scholar]
- PyTorch. Efficientnet_b0. Available online: https://pytorch.org/vision/stable/models/generated/torchvision.models.efficientnet_b0.html#torchvision.models.efficientnet_b0 (accessed on 1 April 2023).
- PyTorch. Vgg16. Available online: https://pytorch.org/vision/stable/models.html#torchvision.models.vgg16 (accessed on 1 April 2023).
- PyTorch. Resnet50. Available online: https://pytorch.org/vision/stable/models.html#torchvision.models.resnet50 (accessed on 1 April 2023).
- PyTorch. Vit_Base_Patch16_224. Available online: https://pytorch.org/vision/stable/models.html#torchvision.models.vit_base_patch16_224 (accessed on 1 April 2023).
- Selvaraju, R.R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; Batra, D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 618–626. [Google Scholar]
- Kanazawa, A.; Uchida, T.; Nakaya, H. Quantitative rainfall intensity estimation based on pixel values in surveillance camera images. Jpn. Soc. Eros. Control. Eng. 2021, 74, 37–48. [Google Scholar]
WNI Indicators | Proposal System | Amount of Rainfall | Criterion |
---|---|---|---|
potsupotsu | Light | Less than 1 mm | No need for an umbrella |
parapara | 1∼2 mm | Umbrellas are needed | |
sa | Medium | 2∼4 mm | Ordinary rain |
zaza | Heavy | 4∼10 mm | Need a large umbrella Difficulty in going out |
goo | More than 10 mm | Downpour Going Out Dangerous |
Class | Amount of Rainfall | Number of Images |
---|---|---|
light | Less than 1 mm | 604 |
1∼2 mm | ||
medium | 2∼4 mm | 687 |
heavy | 4∼10 mm | 604 |
More than 10 mm | ||
Total | 1895 |
Model | Average Acc (%) | Max Acc (%) |
---|---|---|
EfficientNet | 90.09 | 92.61 |
VGG16 | 85.16 | 90.34 |
ResNet50 | 87.37 | 90.91 |
ViT | 62.56 | 67.05 |
Model | Average Acc (%) | Max Acc (%) | F1-Score (%) | ||
---|---|---|---|---|---|
Light | Medium | Heavy | |||
Original: whole-area model | 90.09 | 92.61 | 88 | 91 | 99 |
Proposal: road-area model | 87.25 | 89.2 | 84 | 84 | 100 |
Cloud Name | Symbol | Initial Quantity | Final Quantity | Characteristics |
---|---|---|---|---|
Cumulonimbus | Cb | 242 | 229 | Thunder cloud, icy, anvil-shaped top noted by heavy rain |
Nimbostratus | Ns | 274 | 252 | Rain cloud, grey cloud with a dark and a vague outline |
Cirrus | Ci | 139 | 242 | Feathery, wispy clouds of ice crystals |
Cirrostratus | Cs | 287 | Ice crystals, milky, translucent veil cloud | |
Cirrocumulus | Cc | 268 | White flakes, fleecy clouds forming ripples | |
Altocumulus | Ac | 221 | White or gray with shading and rounded clumps | |
Altostratus | As | 188 | Mainly gray or bluish clouds, opaque and ice crystals | |
Cumulus | Cu | 182 | Cauliflower shape, fluffy, associated by rain or snow showers | |
Stratocumulus | Sc | 340 | Compound dark grey layer cloud rollers or banks | |
Stratus | St | 202 | Low cloud, causes fog, drizzle, or fine precipitation | |
Contrails | Ct | 200 | Aircraft engine exhausts generate these line-shaped clouds | |
Total | 2543 | 723 |
Model | Average Acc (%) | Max Acc (%) |
---|---|---|
EfficientNet | 65.45 | 70.8 |
VGG16 | 49.15 | 54.01 |
ResNet50 | 63.75 | 64.23 |
ViT | 55.35 | 57.66 |
Model | Average Acc (%) | Max Acc (%) | F1-Score (%) | ||
---|---|---|---|---|---|
Cb | Ns | Other | |||
Original: whole-area model | 65.45 | 70.8 | 84 | 60 | 67 |
Proposal: sky-area model | 68.61 | 72.26 | 80 | 69 | 69 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Suemitsu, K.; Endo, S.; Sato, S. Classification of Rainfall Intensity and Cloud Type from Dash Cam Images Using Feature Removal by Masking. Climate 2024, 12, 70. https://doi.org/10.3390/cli12050070
Suemitsu K, Endo S, Sato S. Classification of Rainfall Intensity and Cloud Type from Dash Cam Images Using Feature Removal by Masking. Climate. 2024; 12(5):70. https://doi.org/10.3390/cli12050070
Chicago/Turabian StyleSuemitsu, Kodai, Satoshi Endo, and Shunsuke Sato. 2024. "Classification of Rainfall Intensity and Cloud Type from Dash Cam Images Using Feature Removal by Masking" Climate 12, no. 5: 70. https://doi.org/10.3390/cli12050070