Next Article in Journal
Multi-Temporal Satellite Interferometry for Fast-Motion Detection: An Application to Salt Solution Mining
Previous Article in Journal
Novel Techniques for Void Filling in Glacier Elevation Change Data Sets
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Deep Learning-Based Masonry Wall Image Analysis

1
3in-PPCU Research Group, Péter Pázmány Catholic University, H-2500 Esztergom, Hungary
2
Institute for Computer Science and Control (SZTAKI), H-1111 Budapest, Hungary
3
Faculty of Informatics, University of Debrecen, 4028 Debrecen, Hungary
*
Author to whom correspondence should be addressed.
Remote Sens. 2020, 12(23), 3918; https://doi.org/10.3390/rs12233918
Submission received: 24 October 2020 / Revised: 22 November 2020 / Accepted: 26 November 2020 / Published: 29 November 2020

Abstract

In this paper we introduce a novel machine learning-based fully automatic approach for the semantic analysis and documentation of masonry wall images, performing in parallel automatic detection and virtual completion of occluded or damaged wall regions, and brick segmentation leading to an accurate model of the wall structure. For this purpose, we propose a four-stage algorithm which comprises three interacting deep neural networks and a watershed transform-based brick outline extraction step. At the beginning, a U-Net-based sub-network performs initial wall segmentation into brick, mortar and occluded regions, which is followed by a two-stage adversarial inpainting model. The first adversarial network predicts the schematic mortar-brick pattern of the occluded areas based on the observed wall structure, providing in itself valuable structural information for archeological and architectural applications. The second adversarial network predicts the pixels’ color values yielding a realistic visual experience for the observer. Finally, using the neural network outputs as markers in a watershed-based segmentation process, we generate the accurate contours of the individual bricks, both in the originally visible and in the artificially inpainted wall regions. Note that while the first three stages implement a sequential pipeline, they interact through dependencies of their loss functions admitting the consideration of hidden feature dependencies between the different network components. For training and testing the network a new dataset has been created, and an extensive qualitative and quantitative evaluation versus the state-of-the-art is given. The experiments confirmed that the proposed method outperforms the reference techniques both in terms of wall structure estimation and regarding the visual quality of the inpainting step, moreover it can be robustly used for various different masonry wall types.
Keywords: masonry wall; segmentation; inpainting; U-Net; GANs; watershed transform masonry wall; segmentation; inpainting; U-Net; GANs; watershed transform
Graphical Abstract

Share and Cite

MDPI and ACS Style

Ibrahim, Y.; Nagy, B.; Benedek, C. Deep Learning-Based Masonry Wall Image Analysis. Remote Sens. 2020, 12, 3918. https://doi.org/10.3390/rs12233918

AMA Style

Ibrahim Y, Nagy B, Benedek C. Deep Learning-Based Masonry Wall Image Analysis. Remote Sensing. 2020; 12(23):3918. https://doi.org/10.3390/rs12233918

Chicago/Turabian Style

Ibrahim, Yahya, Balázs Nagy, and Csaba Benedek. 2020. "Deep Learning-Based Masonry Wall Image Analysis" Remote Sensing 12, no. 23: 3918. https://doi.org/10.3390/rs12233918

APA Style

Ibrahim, Y., Nagy, B., & Benedek, C. (2020). Deep Learning-Based Masonry Wall Image Analysis. Remote Sensing, 12(23), 3918. https://doi.org/10.3390/rs12233918

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop