Deep Learning in Data-Driven Pavement Image Analysis and Automated Distress Detection: A Review
Abstract
:1. Introduction
2. Some Existing and Emerging Deep Learning Frameworks for Computer Vision Applications
2.1. Caffe
2.2. TensorFlow
2.3. Theano
2.4. Torch
2.5. Keras
3. Application of Deep Learning to Vision-Based Pavement Distress Detection
3.1. Detecting No-Crack Surfaces from Mobile Mapping Images
3.2. Crack Detection from Low-Cost Smartphone Pavement Images
3.3. Effect of DCNN Depth on Pavement Crack Detection Accuracy
3.4. Generalization of DCNN on Large Open-Source Pavement Distress Dataset
3.5. RoadDamageDetector: A DL Mobile App for Road Damage Detection Based on Open-Source Smartphone Road Images
3.6. Measurement and 3-D Reconstruction of Concealed Cracks in Asphalt Pavements Using GPR Images
3.7. Segmented Grid Based Pavement Crack Classification with DL and PCA
3.8. Learning the Structure of Pavement Cracks from Raw Image Patches
3.9. Continuous Pavement Inspection with CNN Trained on Google Street View Images
3.10. Pixel-Level Crack Detection on 3D Asphalt Pavement Surfaces
3.11. Automated Crack Detection with Pre-Trained DL Model Using Transfer Learning
3.12. Sealed Crack Detection with Transfer Learning and Fine-Tuning
3.13. Other Related Studies
4. Discussion
4.1. Objectives and Datasets
4.2. Network Architecture and Hyper-Parameters
4.3. Software Framework, Hardware Specs, and Test Results Summary
5. Summary, Conclusions, and Future Directions
Funding
Conflicts of Interest
References
- ASCE. American Society of Civil Engineers (ASCE) 2017 Infrastructure Report Card: Roads; American Society of Civil Engineers (ASCE): Reston, VA, USA, 2017. [Google Scholar]
- Flintsch, G.; McGhee, K. NCHRP Synthesis 401: Quality Management of Pavement Condition Data Collection; Transportation Research Board: Washington, DC, USA, 2009. [Google Scholar]
- Gopalakrishnan, K. Advanced Pavement Health Monitoring and Management. IGI Global Videos, 2016. Available online: http://www.igi-global.com/video.aspx?ref=advanced-pavement-health-monitoring-management&titleid=137625 (accessed on 9 May 2017).
- Tsai, Y.; Wang, Z. Development of an Asphalt Pavement Raveling Detection Algorithm Using Emerging 3D Laser Technology and Macrotexture Analysis; Final Report NCHRP IDEA Project 163; Transportation Research Board: Washington, DC, USA, 2015. [Google Scholar]
- Wang, K.C.P.; Li, Q.J.; Yang, G.; Zhan, Y.; Qiu, Y. Network level pavement evaluation with 1 mm 3D survey system. J. Traffic Transp. Eng. Engl. Ed. 2015, 2, 391–398. [Google Scholar] [CrossRef]
- Xie, D.; Zhang, L.; Bai, L. Deep Learning in Visual Computing and Signal Processing. Appl. Comput. Intell. Soft Comput. 2017, 2017, e1320780. [Google Scholar] [CrossRef]
- Agrawal, A.; Choudhary, A. Perspective: Materials informatics and big data: Realization of the ‘fourth paradigm’ of science in materials science. APL Mater. 2016, 4, 053208. [Google Scholar] [CrossRef]
- Liu, W.; Wang, Z.; Liu, X.; Zeng, N.; Liu, Y.; Alsaadi, F.E. A survey of deep neural network architectures and their applications. Neurocomputing 2017, 234, 11–26. [Google Scholar] [CrossRef]
- Bahrampour, S.; Ramakrishnan, N.; Schott, L.; Shah, M. Comparative Study of Deep Learning Software Frameworks. arXiv, 2016; arXiv:1511.06435. [Google Scholar]
- Fonnegra, R.D.; Blair, B.; Díaz, G.M. Performance comparison of deep learning frameworks in image classification problems using convolutional and recurrent networks. In Proceedings of the 2017 IEEE Colombian Conference on Communications and Computing (COLCOM), Cartagena, Colombia, 16–18 August 2017; pp. 1–6. [Google Scholar]
- Shi, S.; Wang, Q.; Xu, P.; Chu, X. Benchmarking State-of-the-Art Deep Learning Software Tools. In Proceedings of the 2016 7th International Conference on Cloud Computing and Big Data (CCBD), Macau, China, 16–18 November 2016; pp. 99–104. [Google Scholar]
- Caffe. Caffe—Deep Learning Framework. 2018. Available online: http://caffe.berkeleyvision.org/ (accessed on 15 February 2018).
- Jia, Y.; Shelhamer, E.; Donahue, J.; Karayev, S.; Long, J.; Girshick, R.; Guadarrama, S.; Darrell, T. Caffe: Convolutional Architecture for Fast Feature Embedding. In Proceedings of the 22nd ACM International Conference on Multimedia, New York, NY, USA, 19–21 August 1998; pp. 675–678. [Google Scholar]
- TensorFlow. TensorFlow. 2018. Available online: https://www.tensorflow.org/ (accessed on 15 February 2018).
- Theano. Theano at a Glance—Theano 1.0.0 Documentation. 2018. Available online: http://deeplearning.net/software/theano/introduction.html (accessed on 16 February 2018).
- Torch. Torch—Scientific Computing for LuaJIT. 2018. Available online: http://torch.ch/ (accessed on 16 February 2018).
- Chollet, F. Keras; GitHub: San Francisco, CA, USA, 2015. [Google Scholar]
- Chambon, S.; Moliard, J.-M. Automatic Road Pavement Assessment with Image Processing: Review and Comparison. Int. J. Geophys. 2011, 20. [Google Scholar] [CrossRef]
- Oliveira, H.; Correia, P.L. Automatic road crack segmentation using entropy and image dynamic thresholding. In Proceedings of the 2009 17th European Signal Processing Conference, Glasgow, UK, 24–28 August 2009; pp. 622–626. [Google Scholar]
- Tsai, Y.-C.; Kaul, V.; Mersereau, R.M. Critical assessment of pavement distress segmentation methods. J. Transp. Eng. 2009, 136, 11–19. [Google Scholar] [CrossRef]
- Zhang, D.; Li, Q.; Chen, Y.; Cao, M.; He, L.; Zhang, B. An efficient and reliable coarse-to-fine approach for asphalt pavement crack detection. Image Vis. Comput. 2017, 57, 130–146. [Google Scholar] [CrossRef]
- Ayenu-Prah, A.; Attoh-Okine, N. Evaluating Pavement Cracks with Bidimensional Empirical Mode Decomposition. EURASIP J. Adv. Signal Process. 2008, 861701, 2008. [Google Scholar] [CrossRef]
- Subirats, P.; Dumoulin, J.; Legeay, V.; Barba, D. Automation of Pavement Surface Crack Detection using the Continuous Wavelet Transform. In Proceedings of the 2006 International Conference on Image Processing, Atlanta, GA, USA, 8–11 October 2006; pp. 3037–3040. [Google Scholar]
- Wang, K.C.P.; Li, Q.; Gong, W. Wavelet-Based Pavement Distress Image Edge Detection with à Trous Algorithm. Transp. Res. Rec. J. Transp. Res. Board 2007, 2024, 73–81. [Google Scholar] [CrossRef]
- Ying, L.; Salari, E. Beamlet Transform-Based Technique for Pavement Crack Detection and Classification. Comput. Aided Civ. Infrastruct. Eng. 2010, 25, 572–580. [Google Scholar] [CrossRef]
- Koch, C.; Georgieva, K.; Kasireddy, V.; Akinci, B.; Fieguth, P. A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure. Adv. Eng. Inform. 2015, 29, 196–210. [Google Scholar] [CrossRef] [Green Version]
- Oliveira, H.; Correia, P.L. Automatic Road Crack Detection and Characterization. IEEE Trans. Intell. Transp. Syst. 2013, 14, 155–168. [Google Scholar] [CrossRef]
- Fujita, Y.; Shimada, K.; Ichihara, M.; Hamamoto, Y. A method based on machine learning using hand-crafted features for crack detection from asphalt pavement surface images. In Proceedings of the Thirteenth International Conference on Quality Control by Artificial Vision 2017, Tokyo, Japan, 14 May 2017; Volume 10338, 103380M. [Google Scholar]
- Hizukuri, A.; Nagata, T. Development of a classification method for a crack on a pavement surface images using machine learning. In Proceedings of the Thirteenth International Conference on Quality Control by Artificial Vision 2017, Tokyo, Japan, 14 May 2017; Volume 10338, 103380M. [Google Scholar]
- Zou, Q.; Cao, Y.; Li, Q.; Mao, Q.; Wang, S. CrackTree: Automatic crack detection from pavement images. Pattern Recognit. Lett. 2012, 33, 227–238. [Google Scholar] [CrossRef]
- Some, L. Automatic Image-Based Road Crack Detection Methods; KTH Royal Institute of Technology: Stockholm, Sweden, 2016. [Google Scholar]
- Oliveira, H.; Correia, P.L. CrackIT—An image processing toolbox for crack detection and characterization. In Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP), Paris, France, 27–30 October 2014; pp. 798–802. [Google Scholar]
- Zhang, L.; Yang, F.; Zhang, Y.D.; Zhu, Y.J. Road crack detection using deep convolutional neural network. In Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 25–28 September 2016; pp. 3708–3712. [Google Scholar]
- Pauly, L.; Peel, H.; Hogg, D.; Fuentes, R. Deeper Networks for Pavement Crack Detection. In Proceedings of the 34th International Symposium on Automation and Robotics in Construction (ISARC 2017), Taipei, Taiwan, 28 June–1 July 2017; pp. 1–7. [Google Scholar]
- Eisenbach, M.; Stricker, R.; Seichter, D.; Amende, K.; Debes, K.; Sesselmann, M.; Ebersbach, D.; Stoeckert, U.; Gross, H.M. How to get pavement distress detection ready for deep learning? A systematic approach. In Proceedings of the 2017 International Joint Conference on Neural Networks (IJCNN), Anchorage, AK, USA, 14–19 May 2017; pp. 2039–2047. [Google Scholar]
- Maeda, H.; Sekimoto, Y.; Seto, T.; Kashiyama, T.; Omata, H. Road Damage Detection Using Deep Neural Networks with Images Captured Through a Smartphone. arXiv, 2018; arXiv:1801.09454. [Google Scholar]
- Maeda, H.; Sekimoto, Y.; Seto, T. Lightweight Road Manager: Smartphone-based Automatic Determination of Road Damage Status by Deep Neural Network. In Proceedings of the 5th ACM SIGSPATIAL International Workshop on Mobile Geographic Information Systems, New York, NY, USA, 31 October 2016; pp. 37–45. [Google Scholar]
- Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, Ch.; Berg, A.C. SSD: Single Shot MultiBox Detector. In European Conference on Computer Vision; Springer: Cham, Switzerland, 2016; Volume 9905, pp. 21–37. [Google Scholar]
- Howard, A.G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; Adam, H. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv, 2017; arXiv:1704.04861. [Google Scholar]
- Dhakal, N.; Elseifi, M.A.; Zhang, Z. Mitigation strategies for reflection cracking in rehabilitated pavements—A synthesis. Int. J. Pavement Res. Technol. 2016, 9, 228–239. [Google Scholar] [CrossRef]
- Tong, Z.; Gao, J.; Zhang, H. Recognition, location, measurement, and 3D reconstruction of concealed cracks using convolutional neural networks. Constr. Build. Mater. 2017, 146, 775–787. [Google Scholar] [CrossRef]
- Wang, X.; Hu, Z. Grid-based pavement crack analysis using deep learning. In Proceedings of the 4th International Conference on Transportation Information and Safety (ICTIS), Banff, AB, Canada, 8–10 August 2017; pp. 917–924. [Google Scholar]
- Fan, Z.; Wu, Y.; Li, W. Automatic Pavement Crack Detection Based on Structured Prediction with the Convolutional Neural Network. arXiv, 2018; arXiv:1802.02208. [Google Scholar]
- Shi, Y.; Cui, L.; Qi, Z.; Meng, F.; Chen, Z. Automatic Road Crack Detection Using Random Structured Forests. IEEE Trans. Intell. Transp. Syst. 2016, 17, 3434–3445. [Google Scholar] [CrossRef]
- Ma, K.; Hoai, M.; Samaras, D. Large-scale Continual Road Inspection: Visual Infrastructure Assessment in the Wild. In Proceedings of the British Machine Vision Conference, London, UK, 4–7 September 2017. [Google Scholar]
- Zhang, A.; Wang, K.C.P.; Li, B.; Yang, E.; Dai, X.; Peng, Y.; Fei, Y.; Liu, Y.; Li, J.Q.; Chen, C. Automated Pixel-Level Pavement Crack Detection on 3D Asphalt Surfaces Using a Deep-Learning Network. Comput. Civ. Infrastruct. Eng. 2017, 32, 805–819. [Google Scholar] [CrossRef]
- Gopalakrishnan, K.; Khaitan, S.K.; Choudhary, A.; Agrawal, A. Deep Convolutional Neural Networks with transfer learning for computer vision-based data-driven pavement distress detection. Constr. Build. Mater. 2017, 157, 322–330. [Google Scholar] [CrossRef]
- Shin, H.-C.; Roth, H.R.; Gao, M.; Lu, L.; Xu, Z.; Nogues, I.; Yao, J.; Mollura, D.; Summers, R.M. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. Med. Imaging 2016, 35, 1285–1298. [Google Scholar] [CrossRef] [PubMed]
- Bar, Y.; Diamant, I.; Wolf, L.; Lieberman, S.; Konen, E.; Greenspan, H. Chest pathology detection using deep learning with non-medical training. In Proceedings of the 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI), New York, NY, USA, 16–19 April 2015; pp. 294–297. [Google Scholar]
- Litjens, G.; Kooi, T.; Bejnordi, B.E.; Setio, A.A.A.; Ciompi, F.; Ghafoorian, M.; van der Laak, J.A.W.M.; van Ginneken, B.; Sánchez, C.I. A survey on deep learning in medical image analysis. Med. Image Anal. 2017, 42, 60–88. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zhang, K.; Cheng, H.D.; Zhang, B. Unified Approach to Pavement Crack and Sealed Crack Detection Using Preclassification Based on Transfer Learning. J. Comput. Civ. Eng. 2018, 32, 04018001. [Google Scholar] [CrossRef]
- Cha, Y.-J.; Choi, W.; Büyüköztürk, O. Deep Learning-Based Crack Damage Detection Using Convolutional Neural Networks. Comput. Aided Civ. Infrastruct. Eng. 2017, 32, 361–378. [Google Scholar] [CrossRef]
- Feng, C.; Liu, M.-Y.; Lee, T.-Y. Deep Active Learning for Civil Infrastructure Defect Detection and Classification. In Proceedings of the 2017 International Workshop on Computing in Civil Engineering, Seattle, WA, USA, 25–27 June 2017. [Google Scholar]
- Gopalakrishnan, K.; Gholami, H.; Vidyadharan, A.; Choudhary, A.; Agrawal, A. Crack Damage Detection in Unmanned Aerial Vehicle Images of Civil Infrastructure Using Pre-Trained Deep Learning Model. Int. J. Traffic Transp. Eng. 2018, 8, 1–14. [Google Scholar]
- Buda, M.; Maki, A.; Mazurowski, M.A. A systematic study of the class imbalance problem in convolutional neural networks. arXiv, 2017; arXiv:1710.05381. [Google Scholar]
- Stanford University. Unsupervised Feature Learning and Deep Learning Tutorial. Deep Learning Tutotrial, 2018. Available online: http://ufldl.stanford.edu/tutorial/selftaughtlearning/SelfTaughtLearning/ (accessed on 9 March 2018).
Reference | Dataset | Goal |
---|---|---|
Some [31] | Street view images (private) | Decrease dataset by detecting no-crack road surfaces (image-level) |
Zhang et al. [33] | Smartphone images (private) | Crack detection from smartphone images (image-level) |
Pauly et al. [34] | Smartphone images [33] | Use of deeper convolutional neural network (CNNs) for pavement crack detection (image-level) |
Eisenbach et al. [35] | German asphalt pavement distress (GAPs) (public) | Generalization of CNN on large open-source pavement distress data (block-level) |
Maeda et al. [36] | Smartphone street view images (public) | A DL mobile app for road damage detection (image-level) |
Tong et al. [41] | GPR images (private) | Location and measurement of concealed cracks in asphalt pavements (block-level) |
Wang and Hu [42] | Smartphone images (private) | Segmented grid based pavement crack classification (block-level) |
Fan et al. [43] | CFD (Shi et al., 2016) [11]; AigleRN (Chambon and Moliard, 2011) [18] (public) | Learning the structure of pavement cracks from raw image patches (pixel-level) |
Ma et al. [45] | Google StreetView images (public) | Continuous pavement inspection “in the wild” (image-level) |
Zhang et al. [21] | 3D asphalt surface images from PaveVision3D system (private) | Pixel-level crack detection on 3D asphalt pavement surfaces (pixel-level) |
Gopalakrishnan et al. [47] | FHWA/LTPP (public) | Crack detection with pre-trained DL model using transfer learning (image-level) |
Zhang et al. [51] | Local (private) | Sealed crack detection with transfer learning and fine tuning (block-level) |
Reference | Network Architecture | Hyper-Parameters |
---|---|---|
Some [31] | CNN: N/A | B = 13; E = 50 |
Zhang et al. [33] | ConvNet (inspired by LeNet): 4 C; 4 MP; 2 FC; 1 O | Op = SGD; R = Dropout (0.5); A = ReLU; B = 48; E = 20; M = 0.9; WD = 0.0005 |
Pauly et al. [34] | ConvNet (inspired by LeNet): 5 C; 4 MP; 2 FC; 1 O | Op = SGD; R = Dropout; A = ReLU; B = 48; E = 40–80; M = 0.9; LR = 0.0001; WD = 0.0005 |
Eisenbach et al. [35] | ASINVOS (inspired by VGG-net and AlexNet): 8 C; 3 MP; 3 FC | Op = SGD; R = Dropout (0.1–0.5); A = ReLU; B = 256; M = 0.7; LR = 0.01 |
Maeda et al. [36] | SSD Inception V2 SSD MobileNet | SSD Inception V2: LR = 0.002 (LR decay = 0.000095) SSD MobileNet: LR = 0.003 (LR decay = 0.000095) |
Tong et al. [41] | Recognition CNN: 2 C; 2 MP; 2 FC; 1 O | Op = Back-propagation; A = Sigmoid; B = 100; E = 30 |
Wang and Hu [42] | CNN: 2 C; 2 MP; 1 FC | A = tanh; B = 32; LR = 0.1 |
Fan et al. [43] | CNN: 4 C; 2 MP; 3 FC | Op = Adam; R = Dropout (0.5); A = ReLU; B = 256; LR = 0.001; WD = 0.0005; E = 13–43 |
Ma et al. [45] | Fisher Vectors with CNN (FV–CNN) using VGG-D (Simonyan and Zisserman, 2014) | N/A |
Zhang et al. [21] | CNN: 2 C; 2 FC; 1 O | Op = MBGD; R = Dropout (0.5); B = 10–30; LR = 0.001–0.05; E = 700 |
Gopalakrishnan et al. [47] | Truncated VGG-16 + transfer learning classifier | Classifier: Op = Adam; R = Dropout (0.5); A = ReLU; B = 32; E = 50 |
Zhang et al. [51] | CUDA-ConvNet (Krizhevsky et al. 2012) + transfer learning: 5 C; 3 FC; 1 O | Op = MBGD; LR = 0.001; WD = 0.005; B = 400; E = 40,000 |
Reference | DL Software Framework | CPU Specs | GPU Specs | Test Results Summary | Method(s) for Comparison |
---|---|---|---|---|---|
Some [31] | DIGITS/Caffe | N/A | N/A | A = 0.9225; P = 0.9841; R = 0.8493 | CrackIT |
Zhang et al. [33] | Caffe | Intel® Xeon® E3-1241 V3 @ 3.5 GHz (8 GB RAM) | NVIDIA Quadro K220 | P = 0.8696; R = 0.9251; F1 = 0.8965 | SVM and Boosting Methods |
Pauly et al. [34] | N/A | Intel® Xeon® E5-1630 v4 @ 3.70 GHz (128GB RAM) | NVIDIA Quadro M4000 | A = 0.913; P = 0.907; R = 0.920 | N/A |
Eisenbach et al. [35] | Keras (Theano backend) | N/A | NVIDIA Titan X | A = 0.9772; F1 = 0.7246 | CrackIT |
Maeda et al. [36] | TensorFlow | N/A | NVIDIA GRID K520 | AC: A = 0.85; P = 0.73; R = 0.68 | N/A |
Tong et al. [41] | Caffe | Intel® CoreTM i7-6700 (8 GB RAM) | NVIDIA GeForce GTX | Recognition: A = 0.998 | N/A |
Wang and Hu [42] | TensorFlow | N/A | N/A | AC: A = 0.901 | Neural Networks |
Fan et al. [43] | TensorFlow | Intel® Xeon® E5-2690 2.9 GHz (64 GB RAM) | NVIDIA Quadro K5000 | P = 0.9018; R = 0.9494; F1 = 0.9210 | Canny; Local Thresholding; CrackForest |
Ma et al. [45] | Keras (TensorFlow backend) | N/A | N/A | Average A = 0.582 | SVM |
Zhang et al. [21] | C++ (CUDA C Platform without using NVIDIA cuDNN library) | N/A | NVIDIA GeForce GTX Titan (2 devices) | P = 0.9013; R = 0.8763; F1 = 0.8886 | Pixel-SVM; Shadow modeling |
Gopalakrishnan et al. [47] | Keras (Theano backend) | Intel® CoreTM i7-5600 @ 2.60 GHz (20 GB RAM) | Not used | A = 0.90; P = 0.90; R = 0.90; F1 = 0.90 | N/A |
Zhang et al. [51] | Caffe with MATLAB | HP Z220 workstation | NVIDIA Quadro K4000 | P = 0.847; R = 0.951; F1 = 0.895 | Canny; CrackIT; CrackForest |
© 2018 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Gopalakrishnan, K. Deep Learning in Data-Driven Pavement Image Analysis and Automated Distress Detection: A Review. Data 2018, 3, 28. https://doi.org/10.3390/data3030028
Gopalakrishnan K. Deep Learning in Data-Driven Pavement Image Analysis and Automated Distress Detection: A Review. Data. 2018; 3(3):28. https://doi.org/10.3390/data3030028
Chicago/Turabian StyleGopalakrishnan, Kasthurirangan. 2018. "Deep Learning in Data-Driven Pavement Image Analysis and Automated Distress Detection: A Review" Data 3, no. 3: 28. https://doi.org/10.3390/data3030028