Development of an Artificial Intelligence Model to Recognise Construction Waste by Applying Image Data Augmentation and Transfer Learning
Abstract
:1. Introduction
2. Related Works
2.1. Waste Management
2.2. Convolutional Neural Network (CNN)
2.3. Comparison of Artificial Intelligence Models
3. Development of Recognition Model for Five Types of Construction Waste
3.1. Development Procedure
3.2. Constructing the Dataset and Selecting the Learning Model
3.3. Constructing the Learning Dataset
3.4. Optimal Data Labelling Method
3.5. Result of Learning
4. Quantitative Evaluation Method for Learning Data Using the Fréchet Inception Distance (FID) Technique
4.1. Fréchet Inception Distance (FID) Technique
4.2. Susceptibility Level of Re-Classified Model Due to Noise, Colour Change, and Others
4.2.1. Noise Change
4.2.2. Adding Blur Effect
4.2.3. Hue and Saturation
4.3. Final Learning Results
5. Conclusions
- Advancement in refinement techniques to list the situation on the model function from the data collection step is needed, and not just labelling objects.
- Labelling was impossible without professional knowledge owing to the characteristics of construction waste. Additionally, supervisors were required to manage refined data because there were many objects that could not be differentiated while labelling.
- When the existing classification techniques are mainstream, it is possible to re-use the collected data for an instance segmentation model.
- Regarding the image data with complicated backgrounds, the precise classification of one category seems to enhance the model performance and decrease resource consumption rather than classifying several categories in one image.
- It was verified that increasing the amount of data indiscriminately worsened the quality of the model. Furthermore, it was necessary to apply quantitative augmentation to the learning data in each category.
- To develop an AI model that recognises construction waste, less data with minimum focus and noise, better the collected data performance. Although it does not have much impact on brightness, such as sunlight, to collect data avoiding time, such as sunrise/sunset, which affects image colour, seems better.
- By increasing the amount of data through augmentation using transfer learning, it was verified that mAP increased by 16%. However, the AI model needs to be redesigned by reflecting the characteristics of construction waste if the performance of the model cannot be acquired.
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- The World Bank. World Bank Open Data. 2021. Available online: https://data.worldbank.org (accessed on 21 November 2021).
- Bank of Korea. Economic Statistics System. 2021. Available online: https://ecos.bok.or.kr/EIndex_en.jsp (accessed on 25 November 2021).
- Intergovernmental Panel on Climate Change, Global warming of 1.5C. 2018. Available online: www.ipcc.ch/sr15/download/#chapter (accessed on 30 November 2021).
- Na, S.; Heo, S.-J.; Han, S. Construction Waste Reduction through Application of Different Structural Systems for the Slab in a Commercial Building: A South Korean Case. Appl. Sci. 2021, 11, 5870. [Google Scholar] [CrossRef]
- Davis, P.; Aziz, F.; Newaz, M.T.; Sher, W.; Simon, L. The classification of construction waste material using a deep convolutional neural network. Autom. Constr. 2021, 122, 103481. [Google Scholar] [CrossRef]
- Hollstein, F.; Cacho, Í.; Arnaiz, S.; Wohllebe, M. Challenges in automatic sorting of construction and demolition waste by hyperspectral imaging. In Advanced Environmental, Chemical, and Biological Sensing Technologies XIII; International Society for Optics and Photonics: Washington, DC, USA, 2017. [Google Scholar]
- Won, J.; Cheng, J.C.; Lee, G. Quantification of construction waste prevented by BIM-based design validation: Case studies in South Korea. Waste Manag. 2016, 49, 170–180. [Google Scholar] [CrossRef]
- Cha, G.-W.; Moon, H.J.; Kim, Y.-C.; Hong, W.-H.; Jeon, G.-Y.; Yoon, Y.R.; Hwang, C.; Hwang, J.-H. Evaluating recycling potential of demolition waste considering building structure types: A study in South Korea. J. Clean. Prod. 2020, 256, 120385. [Google Scholar] [CrossRef]
- Pickin, J.; Randell, P.; Trinh, J.; Grant, B. National Waste Report 2018; Department of the Environment and Energy: Melbourne, Victoria, Australia, 2018. [Google Scholar]
- Kabirifar, K.; Mojtahedi, M.; Wang, C.; Tam, V.W. Construction and demolition waste management contributing factors coupled with reduce, reuse, and recycle strategies for effective waste management: A review. J. Clean. Prod. 2020, 263, 121265. [Google Scholar] [CrossRef]
- Lu, W.; Yuan, H. A framework for understanding waste management studies in construction. Waste Manag. 2011, 31, 1252–1260. [Google Scholar] [CrossRef] [Green Version]
- Akinosho, T.D.; Oyedele, L.O.; Bilal, M.; Ajayi, A.O.; Delgado, M.D.; Akinade, O.O.; Ahmed, A.A. Deep learning in the construction industry: A review of present status and future innovations. J. Build. Eng. 2020, 32, 101827. [Google Scholar] [CrossRef]
- Yang, Q.; Shi, W.; Chen, J.; Lin, W. Deep convolution neural network-based transfer learning method for civil infrastructure crack detection. Autom. Constr. 2020, 116, 103199. [Google Scholar] [CrossRef]
- Bolya, D.; Zhou, C.; Xiao, F.; Lee, Y.J. YOLACT: Real-Time Instance Segmentation. In Proceedings of the 2019 IEEE/CVF In-ternational Conference on Computer Vision (ICCV), Seoul, Korea, 27 October–2 November 2019; pp. 9156–9165. [Google Scholar] [CrossRef] [Green Version]
- Tam, V.W.; Tam, C.M. Waste reduction through incentives: A case study. Build. Res. Inf. 2008, 36, 37–43. [Google Scholar] [CrossRef]
- Edwards, D.; Schelling, J. Municipal waste life cycle assessment: Part 2: Transport analysis and glass case study. Process Saf. Environ. Prot. 1999, 77, 259–274. [Google Scholar] [CrossRef]
- Menegaki, M.; Damigos, D. A review on current situation and challenges of construction and demolition waste management. Curr. Opin. Green Sustain. Chem. 2018, 13, 8–15. [Google Scholar] [CrossRef]
- Islam, R.; Nazifa, T.H.; Yuniarto, A.; Uddin, A.S.; Salmiati, S.; Shahid, S. An empirical study of construction and demolition waste generation and implication of recycling. Waste Manag. 2019, 95, 10–21. [Google Scholar] [CrossRef]
- Picon, A.; Ghita, O.; Iriondo, P.M.; Bereciartua, A.; Whelan, P.F. Automation of waste recycling using hyperspectral image analysis. In Proceedings of the 2010 IEEE 15th Conference on Emerging Technologies & Factory Automation (ETFA 2010), Bilbao, Spain, 13–16 September 2010. [Google Scholar]
- Toğaçar, M.; Ergen, B.; Cömert, Z. Waste classification using AutoEncoder network with integrated feature selection method in convolutional neural network models. Measurement 2020, 153, 107459. [Google Scholar] [CrossRef]
- Mao, W.-L.; Chen, W.-C.; Wang, C.-T.; Lin, Y.-H. Recycling waste classification using optimized convolutional neural network. Resour. Conserv. Recycl. 2021, 164, 105132. [Google Scholar] [CrossRef]
- VJ, A.; Balakrishnan, K.; Rosmi, T.B.; Krishna, K.S.; Sreejith, S.; Subha, T.D. Automatic waste segregator and monitoring system. J. Microcontroller Eng. Appl. 2016, 3, 1–7. [Google Scholar]
- Xiao, W.; Yang, J.; Fang, H.; Zhuang, J.; Ku, Y. A robust classification algorithm for separation of construction waste using NIR hyperspectral system. Waste Manag. 2019, 90, 1–9. [Google Scholar] [CrossRef]
- Adedeji, O.; Wang, Z. Intelligent Waste Classification System Using Deep Learning Convolutional Neural Network. Procedia Manuf. 2019, 35, 607–612. [Google Scholar] [CrossRef]
- Cha, Y.-J.; Choi, W.; Büyüköztürk, O. Deep Learning-Based Crack Damage Detection Using Convolutional Neural Networks. Comput. Civ. Infrastruct. Eng. 2017, 32, 361–378. [Google Scholar] [CrossRef]
- Liang, S.; Gu, Y. A deep convolutional neural network to simultaneously localize and recognize waste types in images. Waste Manag. 2021, 126, 247–257. [Google Scholar] [CrossRef]
- Wang, H. Garbage Recognition and Classification System Based on Convolutional Neural Network VGG16. In Proceedings of the 2020 3rd International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE), Shenzen, China, 24–26 April 2020. [Google Scholar]
- Shin, Y.; Heo, S.; Han, S.; Kim, J.; Na, S. An Image-Based Steel Rebar Size Estimation and Counting Method Using a Convolutional Neural Network Combined with Homography. Buildings 2021, 11, 463. [Google Scholar] [CrossRef]
- Cha, Y.-J.; Choi, W. Vision-Based Concrete Crack Detection Using a Convolutional Neural Network. In Dynamics of Civil Structures; Springer: Berlin, Germany, 2017; Volume 2, pp. 71–73. [Google Scholar]
- Zhang, H.; Yan, X.; Li, H. Ergonomic posture recognition using 3D view-invariant features from single ordinary camera. Autom. Constr. 2018, 94, 1–10. [Google Scholar] [CrossRef]
- Abioye, S.O.; Oyedele, L.O.; Akanbi, L.; Ajayi, A.; Delgado, J.M.D.; Bilal, M.; Akinade, O.O.; Ahmed, A. Artificial intelligence in the construction industry: A review of present status, opportunities and future challenges. J. Build. Eng. 2021, 44, 103299. [Google Scholar] [CrossRef]
- Pan, Y.; Zhang, L. Roles of artificial intelligence in construction engineering and management: A critical review and future trends. Autom. Constr. 2021, 122, 103517. [Google Scholar] [CrossRef]
- Deng, F.; He, Y.; Zhou, S.; Yu, Y.; Cheng, H.; Wu, X. Compressive strength prediction of recycled concrete based on deep learning. Constr. Build. Mater. 2018, 175, 562–569. [Google Scholar] [CrossRef]
- Gopalakrishnan, K.; Khaitan, S.K.; Choudhary, A.; Agrawal, A. Deep Convolutional Neural Networks with transfer learning for computer vision-based data-driven pavement distress detection. Constr. Build. Mater. 2017, 157, 322–330. [Google Scholar] [CrossRef]
- Dung, C.V.; Anh, L.D. Autonomous concrete crack detection using deep fully convolutional neural network. Autom. Constr. 2019, 99, 52–58. [Google Scholar] [CrossRef]
- Koch, C.; Georgieva, K.; Kasireddy, V.; Akinci, B.; Fieguth, P. A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure. Adv. Eng. Informatics 2015, 29, 196–210. [Google Scholar] [CrossRef] [Green Version]
- Son, H.; Seong, H.; Choi, H.; Kim, C. Real-Time Vision-Based Warning System for Prevention of Collisions between Workers and Heavy Equipment. J. Comput. Civ. Eng. 2019, 33, 04019029. [Google Scholar] [CrossRef]
- Spencer, B.F., Jr.; Hoskere, V.; Narazaki, Y. Advances in computer vision-based civil infrastructure inspection and monitoring. Engineering 2019, 5, 199–222. [Google Scholar] [CrossRef]
- Yang, M.; Thung, G. Classification of Trash for Recyclability Status. CS229 Project Report 2016. Available online: https://pdfs.semanticscholar.org/c908/11082924011c73fea6252f42b01af9076f28.pdf (accessed on 30 November 2021).
- Proença, P.F.; Simões, P. TACO: Trash Annotations in Context for Litter Detection. arXiv 2020, arXiv:2003.06975. [Google Scholar]
- Ahmad, K.; Khan, K.; Al-Fuqaha, A. Intelligent Fusion of Deep Features for Improved Waste Classification. IEEE Access 2020, 8, 96495–96504. [Google Scholar] [CrossRef]
- Wang, Z.; Peng, B.; Huang, Y.; Sun, G. Classification for plastic bottles recycling based on image recognition. Waste Manag. 2019, 88, 170–181. [Google Scholar] [CrossRef]
- De Carolis, B.; Ladogana, F.; Macchiarulo, N. YOLO TrashNet: Garbage Detection in Video Streams. In Proceedings of the 2020 IEEE Conference on Evolving and Adaptive Intelligent Systems (EAIS), Bari, Italy, 27-29 May 2020. [Google Scholar]
- National Information Society Agency. AI Learning Data Quality Management Guidelines; National Information Society Agency: Daegu, Korea, 2021.
- National Information Society Agency. AI Dataset Construction Guidebook; National Information Society Agency: Daegu, Korea, 2021.
- Ministry of Environment. Enforcement Decree of the Wastes Control Act; Ministry of Environment: Seoul, Korea, 2019.
- Russell, B.C.; Torralba, A.; Murphy, K.P.; Freeman, W.T. LabelMe: A Database and Web-Based Tool for Image Annotation. Int. J. Comput. Vis. 2008, 77, 157–173. [Google Scholar] [CrossRef]
- Heo, S.; Han, S.; Shin, Y.; Na, S. Challenges of Data Refining Process during the Artificial Intelligence Development Projects in the Architecture, Engineering and Construction Industry. Appl. Sci. 2021, 11, 10919. [Google Scholar] [CrossRef]
Super Category | Labelling Method | Amount of Time | Step | Manpower/Hour | Working Time | Metrics per Data | Average |
---|---|---|---|---|---|---|---|
Brick | Segmentation | 112 | Acquisition-cleansing | 2 | 6 | 9.3 | 7.89 |
Labelling | 1 | 31 | 3.6 | ||||
Concrete | 113 | Acquisition-cleansing | 2 | 6 | 9.4 | ||
Labelling | 1 | 32 | 3.5 | ||||
Wood | 139 | Acquisition-cleansing | 2 | 6 | 3.5 | ||
Labelling | 2 | 11 | 6.3 | ||||
Board | 129 | Acquisition-cleansing | 2 | 6 | 10.7 | ||
Labelling | 1 | 23 | 5.6 | ||||
Mixed bag | 158 | Acquisition-cleansing | 2 | 6 | 13.2 | ||
Labelling | 1 | 22 | 7.1 |
Case | Category | Quantity | Labelling Method | Work Hour (Work Index) | mAP | Result |
---|---|---|---|---|---|---|
A | Class 1 | 100 | Labelling of one object with one category using a clean background | 1 man/40 h (2.5) | 34 | Masking was formed generally in a good shape with a waste boundary |
B | 100 | Labelling of one category in an image with various objects | 1 man/4 h (2.5) | 24 | Able to classify but unable to recognise some complex images | |
C | Class 5 | 153 | Labelling by classifying all five categories in an image with various objects | 4 men/22 h (1.73) | 33 | Well recognised, but experienced confusion in most classes and could not follow the boundary |
D | 153 | Labelling by classifying one category in an image with various objects | 4 men/16 h (2.39) | 39 | Generally, well recognised and experienced confusion with a type of class, but followed the boundary well |
Iteration 10k | Images | mAP | ||||||
---|---|---|---|---|---|---|---|---|
All | 0.55 | 0.65 | 0.75. | 0.85 | 0.95 | |||
1000 | Box | 788 | 17.88 | 30.5 | 28.1 | 16.3 | 111.1 | 3.4 |
Mask | 10.24 | 29.1 | 2.4 | 14.6 | 7.1 | 0.5 | ||
5000 | Box | 27.40 | 50.2 | 37.4 | 27.1 | 17.2 | 5.1 | |
Mask | 24.64 | 52.3 | 35.2 | 25.1 | 9.9 | 0.7 | ||
10,000 | Box | 33.90 | 58.4 | 40.5 | 31.1 | 18.4 | 5.6 | |
Mask | 32.50 | 59.3 | 42.4 | 28.4 | 11.3 | 0.8 |
Category | Considered Factors | Step | Predicted Problems | Solutions |
---|---|---|---|---|
Concrete | Crushed concrete was labelled in one mass | Refinement | Recognised sand/object chunk of the floor that are not concrete waste | Differentiates floor/crushed concrete |
Brick | Cement bricks were not photographed | Collection | Unable to differentiate the cement bricks, which had the same colour as that of concrete | Differentiates red brick and concrete |
Wood | Broken cross section was photographed | Collection | Only recognized objects in length | Able to differentiate short or side wood |
Board | Broken board was not labelled | Refinement | Broken board was misrecognised as concrete or brick waste | Differentiates relatively shaped boards |
Mixed bag | Contents inside a waste bag were not labelled | Refinement | Recognition error as other waste | Exactly differentiates only the waste bags |
Number of Times | Image Quality | Iteration 10,000 | All | 0.55 | 0.65 | 0.75 | 0.85 | 0.95 |
---|---|---|---|---|---|---|---|---|
0 | 788 | Box | 33.9 | 58.4 | 40.5 | 31.1 | 18.4 | 5.6 |
Mask | 32.5 | 59.3 | 42.4 | 28.4 | 11.3 | 0.8 | ||
100 | 1576 | Box | 36.2 | 63.2 | 44.2 | 39.2 | 26.1 | 8.3 |
Mask | 32.4 | 62.1 | 46.3 | 33.2 | 18.4 | 2.2 | ||
150 | 1576 | Box | 30.9 | 54.3 | 39.2 | 33.1 | 17.2 | 6.6 |
Mask | 28.2 | 57.6 | 44.2 | 26.3 | 11.7 | 1.4 | ||
200 | 1576 | Box | 27.6 | 52.1 | 38.2 | 28.9 | 16.3 | 2.4 |
Mask | 24.7 | 48.2 | 42.5 | 24.3 | 8.2 | 1.4 | ||
250 | 1576 | Box | 15.3 | 34.2 | 22.4 | 12.2 | 6.5 | 1.2 |
Mask | 14.8 | 33.7 | 23.2 | 10.4 | 6.3 | 0.5 |
Sigma | Image Quantity | Iteration 10,000 | All | 0.55 | 0.65 | 0.75 | 0.85 | 0.95 |
---|---|---|---|---|---|---|---|---|
0 | 788 | Box | 33.9 | 58.4 | 40.5 | 31.1 | 18.4 | 5.6 |
Mask | 32.5 | 59.3 | 42.4 | 28.4 | 11.3 | 0.8 | ||
2 | 1576 | Box | 32.5 | 59.2 | 39.3 | 33.1 | 20.3 | 10.4 |
Mask | 31.5 | 58.4 | 45.2 | 34.1 | 16.2 | 3.4 | ||
4 | 1576 | Box | 15.8 | 33.5 | 23.1 | 11.5 | 8.8 | 2.1 |
Mask | 12.7 | 29.3 | 21.1 | 9.7 | 3.2 | 0.2 | ||
6 | 1576 | Box | 16.0 | 32.9 | 20.9 | 12.3 | 10.2 | 3.9 |
Mask | 13.7 | 29.9 | 23.5 | 10.8 | 4.2 | 0.3 | ||
8 | 1576 | Box | 13.4 | 32.3 | 18.8 | 12.1 | 7.2 | 2.1 |
Mask | 11.0 | 23.0 | 20.1 | 8.3 | 3.3 | 0.2 |
Colour Code Angle | Image Quantity | Iteration 10,000 | All | 0.55 | 0.65 | 0.75 | 0.85 | 0.95 |
---|---|---|---|---|---|---|---|---|
Standard | 788 | Box | 33.9 | 58.4 | 40.5 | 31.1 | 18.4 | 5.6 |
Mask | 32.5 | 59.3 | 42.4 | 28.4 | 11.3 | 0.8 | ||
Hue −20 | 1576 | Box | 32.4 | 59.9 | 41.2 | 33.7 | 20.1 | 7.1 |
Mask | 27.7 | 59.2 | 40.3 | 28.1 | 10.0 | 0.7 | ||
Hue −40 | 1576 | Box | 14.1 | 23.1 | 24.3 | 16.3 | 5.2 | 1.4 |
Mask | 11.9 | 16.2 | 21.5 | 15.3 | 6.2 | 0.5 | ||
Hue −60 (almost black and white) | 1576 | Box | 11.3 | 17.7 | 15.4 | 10.3 | 8.8 | 4.2 |
Mask | 10.2 | 20.3 | 17.3 | 8.3 | 4.5 | 0.4 | ||
Saturation +20 | 1576 | Box | 21.8 | 40.3 | 32.4 | 21.8 | 11.4 | 3.2 |
Mask | 19.0 | 39.4 | 29.4 | 20.5 | 5.2 | 0.3 | ||
Saturation +60 | 1576 | Box | 14.0 | 23.1 | 18.4 | 14.1 | 10.2 | 4.1 |
Mask | 8.9 | 19.3 | 10.4 | 8.2 | 6.1 | 0.3 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Na, S.; Heo, S.; Han, S.; Shin, Y.; Lee, M. Development of an Artificial Intelligence Model to Recognise Construction Waste by Applying Image Data Augmentation and Transfer Learning. Buildings 2022, 12, 175. https://doi.org/10.3390/buildings12020175
Na S, Heo S, Han S, Shin Y, Lee M. Development of an Artificial Intelligence Model to Recognise Construction Waste by Applying Image Data Augmentation and Transfer Learning. Buildings. 2022; 12(2):175. https://doi.org/10.3390/buildings12020175
Chicago/Turabian StyleNa, Seunguk, Seokjae Heo, Sehee Han, Yoonsoo Shin, and Myeunghun Lee. 2022. "Development of an Artificial Intelligence Model to Recognise Construction Waste by Applying Image Data Augmentation and Transfer Learning" Buildings 12, no. 2: 175. https://doi.org/10.3390/buildings12020175