VDCrackGAN: A Generative Adversarial Network with Transformer for Pavement Crack Data Augmentation
Abstract
:1. Introduction
- Based on the idea of VAE-GAN, we propose a generative network for pavement crack data augmentation combining VAE and DCGAN [10]; network architecture was proposed, which integrates the advantages of VAE and DCGAN.
- We introduce spectral normalization [11] to improve the training stability of the network and integrate the self-attention mechanism SwinTransformer into the CNN architecture to improve the generation quality of the generative network.
- Compared with the original DCGAN, VDCrackGAN achieves significant performance improvements of 13.6% and 26.4% in the metrics of Inception Score (IS) and Fréchet Inception Distance (FID), respectively.
2. Related Works
2.1. Data Augmentation
2.2. Generative Adversarial Network and Its Application in Crack Detection
3. Methods
3.1. Network Architecture
3.1.1. Encoder
3.1.2. Generator
3.1.3. Discriminator Network Architecture
3.2. Loss Function
3.2.1. Loss Function of VAE
3.2.2. Loss Function of VDCrackGAN
3.3. Further Network Optimization
3.3.1. Spectral Normalization
3.3.2. Swin Transformer
4. Experiment
4.1. Evaluation Metrics
4.2. Dataset and Experimental Environment
4.3. Experiment Results
4.3.1. Model Comparison Experiment
4.3.2. Ablation Study
The Influence of Different Swin Transformer Block Locations
The Influence of the Number of Heads in Swin Transformer’s Multi-Head Attention
The Influence of the Weight of Each Loss Function
4.3.3. Crack Generating Effect
- We directly compared the human visual effects of the crack samples generated by VDCrakGAN with the real crack images;
- Crack samples generated by VDCrakGAN were used to train the image classification network to evaluate the effectiveness of VDCrakGAN for crack data augmentation.
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Zakeri, H.; Nejad, F.M.; Fahimifar, A. Image Based Techniques for Crack Detection, Classification and Quantification in As-phalt Pavement: A Review. Arch. Comput. Methods Eng. 2017, 24, 935–977. [Google Scholar] [CrossRef]
- Lecun, Y.; Bengio, Y.; Hinton, G. Deep Learning. Nature 2015, 521, 436–444. [Google Scholar] [CrossRef]
- Schmidhuber, J. Deep Learning in Neural Networks: An Overview. Neural Netw. 2015, 61, 85–117. [Google Scholar] [CrossRef] [PubMed]
- Cha, Y.J.; Choi, W.; Büyüköztürk, O. Deep Learning-Based Crack Damage Detection Using Convolutional Neural Networks. Comput. Civ. Infrastruct. Eng. 2017, 32, 361–378. [Google Scholar] [CrossRef]
- Zou, Q.; Zhang, Z.; Li, Q.; Qi, X.; Wang, Q.; Wang, S. DeepCrack: Learning Hierarchical Convolutional Features for Crack Detection. IEEE Trans. Image Process. 2019, 28, 1498–1512. [Google Scholar] [CrossRef]
- Doersch, C. Tutorial on Variational Autoencoders. arXiv 2016, arXiv:1606.05908. [Google Scholar]
- Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative Adversarial Networks. Commun. ACM 2020, 63, 139–144. [Google Scholar] [CrossRef]
- Larsen, A.B.L.; Sønderby, S.K.; Larochelle, H.; Winther, O. Autoencoding beyond Pixels Using a Learned Similarity Metric. In Proceedings of the 33rd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, 19–24 June 2016; Volume 4, pp. 2341–2349. [Google Scholar]
- Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; Guo, B. Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, BC, Canada, 11–17 October 2021; pp. 9992–10002. [Google Scholar] [CrossRef]
- Radford, A.; Metz, L.; Chintala, S. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. In Proceedings of the 4th International Conference on Learning Representations, ICLR 2016-Conference Track Proceedings, San Juan, Puerto Rico, 2–4 May 2016; pp. 1–16. [Google Scholar]
- Miyato, T.; Kataoka, T.; Koyama, M.; Yoshida, Y. Spectral Normalization for Generative Adversarial Networks. In Proceedings of the 6th International Conference on Learning Representations, ICLR 2018-Conference Track Proceedings, Vancouver, BC, Canada, 30 April–3 May 2018. [Google Scholar]
- Yu, X.; Wu, X.; Luo, C.; Ren, P. Deep Learning in Remote Sensing Scene Classification: A Data Augmentation Enhanced Convolutional Neural Network Framework. GIScience Remote Sens. 2017, 54, 741–758. [Google Scholar] [CrossRef]
- González, R.E.; Muñoz, R.P.; Hernández, C.A. Galaxy Detection and Identification Using Deep Learning and Data Augmentation. Astron. Comput. 2018, 25, 103–109. [Google Scholar] [CrossRef]
- Huang, H.; Li, Q.; Zhang, D. Deep Learning Based Image Recognition for Crack and Leakage Defects of Metro Shield Tunnel. Tunn. Undergr. Space Technol. 2018, 77, 166–176. [Google Scholar] [CrossRef]
- Zhang, H.; Cisse, M.; Dauphin, Y.N.; Lopez-paz, D. Mixup: Beyond Empirical Risk Minimization. arXiv 2017, arXiv:1710.09412. [Google Scholar]
- Devries, T.; Taylor, G.W. Improved Regularization of Convolutional Neural Networks with Cutout. arXiv 2017, arXiv:1708.04552. [Google Scholar]
- Yun, S. CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea, 27 October–2 November 2019; pp. 6022–6031. [Google Scholar] [CrossRef]
- Diaz-Pinto, A.; Colomer, A.; Naranjo, V.; Morales, S.; Xu, Y.; Frangi, A.F. Retinal Image Synthesis and Semi-Supervised Learning for Glaucoma Assessment. IEEE Trans. Med. Imaging 2019, 38, 2211–2218. [Google Scholar] [CrossRef] [PubMed]
- Chen, J.; Song, W. GAN-VAE: Elevate Generative Ineffective Image Through Variational Autoencoder. In Proceedings of the 2022 5th International Conference on Pattern Recognition and Artificial Intelligence, PRAI 2022, Chengdu, China, 19–21 August 2022; pp. 765–770. [Google Scholar] [CrossRef]
- Mirza, M.; Osindero, S. Conditional Generative Adversarial Nets. arXiv 2014, arXiv:1411.1784. [Google Scholar]
- Arjovsky, M.; Chintala, S.; Bottou, L. Wasserstein GAN. arXiv 2017, arXiv:1701.07875. [Google Scholar]
- Gulrajani, I.; Ahmed, F.; Arjovsky, M.; Dumoulin, V.; Courville, A. Improved Training of Wasserstein GANs. Adv. Neural Inf. Process. Syst. 2017, 30, 5768–5778. [Google Scholar]
- Zhu, J.; Park, T.; Efros, A.A.; Ai, B.; Berkeley, U.C. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 2242–2251. [Google Scholar] [CrossRef]
- Zhang, H.; Goodfellow, I.; Metaxas, D.; Odena, A. Self-Attention Generative Adversarial Networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, Long Beach, CA, USA, 9–15 June 2019; pp. 12744–12753. [Google Scholar]
- Chen, G.; Teng, S.; Lin, M.; Yang, X.; Sun, X. Crack Detection Based on Generative Adversarial Networks and Deep Learning. KSCE J. Civ. Eng. 2022, 26, 1803–1816. [Google Scholar] [CrossRef]
- Ma, D.; Fang, H.; Wang, N.; Zhang, C.; Dong, J.; Hu, H. Automatic Detection and Counting System for Pavement Cracks Based on PCGAN and YOLO-MF. IEEE Trans. Intell. Transp. Syst. 2022, 23, 22166–22178. [Google Scholar] [CrossRef]
- Hou, Y.; Liu, S.; Cao, D.; Peng, B.; Liu, Z.; Sun, W.; Chen, N. A Deep Learning Method for Pavement Crack Identification Based on Limited Field Images. IEEE Trans. Intell. Transp. Syst. 2022, 23, 22156–22165. [Google Scholar] [CrossRef]
- Maeda, H.; Kashiyama, T.; Sekimoto, Y.; Seto, T.; Omata, H. Generative Adversarial Network for Road Damage Detection. Comput.-Aided Civ. Infrastruct. Eng. 2021, 36, 47–60. [Google Scholar] [CrossRef]
- Jin, T.; Ye, X.W.; Li, Z.X. Establishment and Evaluation of Conditional GAN-Based Image Dataset for Semantic Segmentation of Structural Cracks. Eng. Struct. 2023, 285, 116058. [Google Scholar] [CrossRef]
- Zhang, X.; Peng, B.; Al-Huda, Z.; Zhai, D. FeatureGAN: Combining GAN and Autoencoder for Pavement Crack Image Data Augmentations. Int. J. Image Graph. Signal Process. 2022, 14, 28–43. [Google Scholar] [CrossRef]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016. [Google Scholar]
- Zhang, L.; Yang, F.; Daniel Zhang, Y.; Zhu, Y.J. Road Crack Detection Using Deep Convolutional Neural Network. In Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 25–28 September 2016; pp. 3708–3712. [Google Scholar] [CrossRef]
Layer | Type | Filter Size | Stride | Out Channels | Feature Map Size | Activation |
---|---|---|---|---|---|---|
1 | Conv | 7 × 7 | 2 | 64 | 128 × 128 | ReLu |
2 | Maxpool | 3 × 3 | 2 | 64 | 64 × 64 | ReLu |
3 | ResBlock1 | 3 × 3 | 1 | 64 | 64 × 64 | ReLu |
4 | ResBlock2 | 3 × 3 | 2 | 128 | 32 × 32 | ReLu |
5 | ResBlock3 | 3 × 3 | 2 | 256 | 16 × 16 | ReLu |
6 | ResBlock4 | 3 × 3 | 2 | 512 | 8 × 8 | ReLu |
7 | avgpool | 512 | 1 × 1 | |||
8 | fc(μ) * | 100 | ||||
9 | fc(σ) * | 100 |
Layer | Type | Filter Size | Stride | Out Channels | Feature Map Size | Activation |
---|---|---|---|---|---|---|
1 | Deconv | 4 × 4 | 1 | 512 | 4 × 4 | ReLU |
2 | Deconv | 4 × 4 | 2 | 384 | 8 × 8 | ReLU |
3 | Deconv | 4 × 4 | 2 | 256 | 16 × 16 | ReLU |
4 | Deconv | 4 × 4 | 2 | 192 | 32 × 32 | ReLU |
5 | Deconv | 4 × 4 | 2 | 96 | 64 × 64 | ReLU |
6 | Deconv | 4 × 4 | 2 | 64 | 128 × 128 | ReLU |
7 | Swin Transformer Block | — | — | — | 128 × 128 | — |
8 | Deconv | 4 × 4 | 2 | 3 | 256 × 256 | Tanh |
Layer | Type | Filter Size | Stride | Out Channels | Feature Map Size | Activation |
---|---|---|---|---|---|---|
1 | Conv | 4 × 4 | 2 | 64 | 128 × 128 | LeakyReLU |
2 | Conv | 4 × 4 | 2 | 128 | 64 × 64 | LeakyReLU |
3 | Conv | 4 × 4 | 2 | 256 | 32 × 32 | LeakyReLU |
4 | Conv | 4 × 4 | 2 | 384 | 16 × 16 | LeakyReLU |
5 | Conv | 4 × 4 | 2 | 512 | 8 × 8 | LeakyReLU |
6 | Conv | 4 × 4 | 2 | 768 | 4 × 4 | LeakyReLU |
7 | Conv | 4 × 4 | 1 | 1 | 1 × 1 | Sigmoid |
Type | Configuration |
---|---|
CPU | Intel i9-10900X 3.7 GHz |
GPU | NVIDIA GeForce RTX 3090 24 GB |
RAM | DDR4 3000 MHz 64 GB |
Soft | Python 3.7.6, Pytorch 1.9.0, Cuda 11.3 |
Models | IS | FID | ||
---|---|---|---|---|
Score | std | Score | std | |
real crack dataset | 3.259 | 0.216 | — | — |
SAGAN | 3.212 | 0.036 | 27.139 | 0.075 |
WGAN | 3.176 | 0.048 | 38.762 | 0.092 |
DCGAN | 2.923 | 0.037 | 36.414 | 0.085 |
VAE-DCGAN | 3.106 (↑8.0 *) | 0.046 | 32.331 (↑11.2) | 0.081 |
VAE-DCGAN +SN | 3.146 (↑9.7) | 0.039 | 31.751 (↑12.8) | 0.081 |
VAE-DCGAN +SN + Swin Transformer (our VDCrackGAN) | 3.235 (↑13.6) | 0.040 | 26.812 (↑26.4) | 0.072 |
Position of Swin Transformer Block | IS | FID | ||
---|---|---|---|---|
Score | std | Score | std | |
real crack dataset | 3.259 | 0.216 | — | — |
feat128 1 | 3.235 | 0.040 | 26.812 | 0.072 |
feat64 | 3.162 | 0.038 | 46.410 | 0.097 |
feat32 | 3.214 | 0.046 | 37.333 | 0.086 |
feat16 | 2.995 | 0.033 | 60.955 | 0.091 |
feat8 | 3.148 | 0.038 | 38.033 | 0.083 |
Num_Heads | IS | FID | ||
---|---|---|---|---|
Score | std | Score | std | |
4 | 3.235 | 0.040 | 26.812 | 0.072 |
8 | 3.044 | 0.036 | 46.383 | 0.099 |
16 | 3.006 | 0.033 | 49.501 | 0.101 |
Weight of Loss Function | IS | FID | |||
---|---|---|---|---|---|
Score | std | Score | std | ||
1 | 3.222 | 0.036 | 30.142 | 0.078 | |
1.5 | 3.235 | 0.040 | 26.812 | 0.072 | |
2 | 3.203 | 0.045 | 29.325 | 0.075 | |
3 | 3.112 | 0.037 | 37.858 | 0.087 | |
5 | 3.232 | 0.038 | 30.831 | 0.078 | |
0.01 | 3.235 | 0.040 | 26.812 | 0.072 | |
0.02 | 3.007 | 0.036 | 37.258 | 0.086 | |
0.05 | 2.893 | 0.042 | 73.874 | 0.130 | |
1 | 3.235 | 0.040 | 26.812 | 0.072 | |
1.5 | 3.199 | 0.040 | 41.600 | 0.089 | |
2 | 3.144 | 0.039 | 43.460 | 0.094 | |
5 | 3.159 | 0.038 | 72.186 | 0.116 |
Experiment No. | Training Dataset | Accuracy |
---|---|---|
1 | 500 real cracks | 0.9583 |
2 | 500 real cracks + 1000 generated crack | 0.9667 |
3 | 500 real cracks + 2000 generated cracks | 0.9732 |
4 | 1000 real cracks | 0.9655 |
5 | 1000 real cracks + 1000 generated crack | 0.9735 |
6 | 1000 real cracks+ 2000 generated cracks | 0.9788 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Yu, G.; Zhou, X.; Chen, X. VDCrackGAN: A Generative Adversarial Network with Transformer for Pavement Crack Data Augmentation. Appl. Sci. 2024, 14, 7907. https://doi.org/10.3390/app14177907
Yu G, Zhou X, Chen X. VDCrackGAN: A Generative Adversarial Network with Transformer for Pavement Crack Data Augmentation. Applied Sciences. 2024; 14(17):7907. https://doi.org/10.3390/app14177907
Chicago/Turabian StyleYu, Gui, Xinglin Zhou, and Xiaolan Chen. 2024. "VDCrackGAN: A Generative Adversarial Network with Transformer for Pavement Crack Data Augmentation" Applied Sciences 14, no. 17: 7907. https://doi.org/10.3390/app14177907