Orthofaçade-Based Assisted Inspection Method for Buildings

Featured Application

The method proposed in this paper is suitable for application in the condition assessment of inaccessible building façades or high-rise and large structures of all kinds.

Abstract

Building façade assessment could be performed in a more efficient way using a multidisciplinary approach and modern technologies. This study proposes the orthofaçade-based assisted inspection method (AIM), universal and applicable to different types of façade cladding and suitable for application in the condition assessment of inaccessible building façades or high-rise and large structures of all kinds. The AIM method offers a multidisciplinary approach by combining unmanned aerial vehicle (UAV) technology, electronic tachymetry, and digital image processing techniques (photogrammetry and open-source computer vision methods). The method was verified in a case study performed on a high-rise building façade. On-site data acquisition of high-resolution images of façade and control points was conducted by UAV and tachymetry. The data were further processed in photogrammetric software in order to generate a georeferenced orthofaçade. Crack detection was performed at pixel level via computer code using the OpenCV library methods. The established diagnostic model, defined by control points, enables precise determination of crack location. Crack length, width, or area could be calculated based on the coordinates of its points, by performing simple mathematical operations. The AIM method provides automation of crack detection and precise determination of location and geometrical parameters of detected crack.

1. Introduction

Building inspection and performance monitoring represent the basis for informed decision making regarding maintenance, repair, refurbishment, or other actions during the exploitation stage of a building’s life cycle [1,2]. Each building material and component is subject to gradual degradation as a result of the influence of external and internal destructive factors. One of the most common and most important indicators of the effects of adverse impacts on a building is the appearance of cracks on the façade [3,4], even during or shortly after the construction process [5].

In recent years, the use of modern technologies in the process of inspection and damage assessment of the building’s façade became one of the most researched topics in the international scientific community. There are different ideas on how these technologies can be further developed in order to increase their accuracy and performance [2,6]. The conventional survey method comprises crack detection and quantification of crack geometry, conducted on site through visual inspection by a survey expert. In addition to the result being highly dependent on the individual expertise and experience of the surveyor, this method is also time consuming, expensive, and leads to data redundancy. Furthermore, if the inspection involves work at heights or other occupational hazards, it could lead to tragic consequences [7]. Contemporary surveying technologies, such as terrestrial laser scanning and digital photogrammetry, are faster, more reliable, objective, and accurate, so their use is fully justified. For high-rise buildings survey, unmanned aerial vehicles (UAVs or drones) are recommended, as their use overcomes spatial, temporal, and safety limitations typically related to the traditional inspection of these buildings [2,8].

The application of precise image acquisition techniques represents a fundamental aspect of further image processing [9]. The use of drones for obtaining geoinformation in the form of photographs and spatial data for planning and documenting reconstruction works can be an efficient way of working almost in real time. Even the simplest models of UAVs are equipped with good-quality sensors [10] which enable the collecting of high-resolution images of façade elements. The collected images can be further processed in photogrammetric software in order to generate one large orthomosaic (true orthophoto) image of the entire façade—an orthofaçade. The resulting orthofaçade allows detailed visual inspection of the façade by analysing a single image, as well as damage mapping.

In the case of high-rise buildings, a special difficulty with traditional (eye) image inspection is the size of the orthofaçade image, as it requires high-performance computer resources and a lot of experts’ time to review. Moreover, especially vague and small cracks are difficult to recognize. Automating image-based crack detection would improve the process of visual inspection, especially in terms of efficiency and accuracy. Automation would save time and eliminate errors in the subjective assessment.

In the past decade, techniques for automatic crack identification based on image analysis have been the subject of extensive research [11]. The techniques can be classified into two categories: (1) image processing and (2) machine learning. Both approaches have their advantages and limitations and more research is needed in order to improve their performance and accuracy level. Most of the used techniques address crack detection, without considering the geometric parameters of detected cracks. In addition, there is a limited number of studies related to the automatic detection of façade cracks based on UAV images, whereby the detection is mainly based on individual images [3,12,13,14] or in some cases on one large, stitched façade image [15,16]. Image stitching is generally recommended only for small datasets, while for large datasets, orthomosaic generation should be used [17]. To obtain cracks’ location information, previous research [16] used data from UAV sensors. However, drones with a standard global navigation satellite systems (GNSS) receiver provide image geolocation with an expected accuracy of a few meters [18] and in urban areas, the signal is often interrupted by the roof and/or wall of the buildings [8]. The accuracy in the identification of the crack location can be increased (from meters to centimetres) by the use of ground control points (GCPs) through the georeferencing process of collected UAV images.

The paper proposes an Assisted Inspection Method (AIM) developed for the purpose of crack inspection of a building façade, based on the automated analysis of the georeferenced orthofaçade. To the best knowledge of the authors, in the domain of building façade, crack inspection from a UAV-based georeferenced orthofaçade has not been presented before. In the domain of concrete bridge inspection, several studies have focused on crack analysis from UAV-based images [19,20] or video [21]. In this research, the image processing technique was selected over the machine learning approach, because it works simultaneously to detect the cracks with their attributes [22], as well as because of its availability and simplicity of implementation as it does not require an extensively labelled image dataset containing surface cracks to train the model.

The method offers a multidisciplinary approach by combining UAV technology, electronic tachymetry and digital image processing techniques (digital photogrammetry and open-source computer vision methods). UAV technology and electronic tachymetry are used for on-site data acquisition. The collected data are further processed in photogrammetric software in order to generate an orthofaçade that is geometrically corrected and can be used to measure the true distances of features within the photograph. In the final phase, the generated orthofaçade is used as the data source for crack inspection which includes crack detection and the determination of location and geometrical parameters of detected cracks. Crack detection is performed at the pixel level via computer code using OpenCV library methods [23]. The accuracy of crack detection depends on the achieved GSD (ground sample distance) value of the generated orthofaçade, which corresponds to the pixel size and theoretically represents the smallest detectable crack width. After detection, the proposed diagnostic model, defined by control points, enables the determination of crack location information in a selected coordinate system. The accuracy of spatial positioning of a crack on an orthofaçade corresponds to the generated RMS (root mean square) error of control points. The possibility to address the recognized defect is especially important due to the difficult access to certain elements and the unambiguous addressing of problems during the façade renovation process. In the end, geometrical parameters of a crack (length, width, area) could be obtained based on the coordinates of its points, by performing simple mathematical operations.

The AIM incorporates technologies often applied in contemporary engineering practice and verified open-access digital tools, enabling a simple and easy introduction into the everyday engineering practice of building façade assessment.

2. Materials and Methods

The proposed AIM consists of four main phases (Figure 1):

• Preparation and organisation of inspection;
• Building façade survey;
• Orthofaçade generation;
• Crack inspection.

2.1. Preparation and Organisation of Inspection

The first phase of AIM includes activities related to the preparation and organization of the inspection process. The activities are: (i) analysis of available building design documentation and (ii) UAV mission planning.

Analysis of available building design documentation is the first step carried out in order to collect basic data on the building of interest.

UAV mission planning refers to the planning of the image dataset acquisition process by defining optimal flight parameters. As there is no standard procedure for digital photogrammetry in the building survey, flight parameters are defined on the basis of set objectives (acquired GSD), performances of available equipment and analysis of potential limitations that may arise during the flight in situ, as well as a review of the literature. Defined flight parameters include flight mode, camera orientation, flying distance from the façade, flying path and image overlap. Similar studies were conducted in manual flight mode, with flying distances of 4 m to 13 m from the façade, and overlapping from 60% to 90% [8,10,24,25]. In general cases, the recommended frontal overlap (with respect to the flight direction) is at least 75% and side overlap (between flying tracks) is at least 60% [26]. For the purpose of creating orthophotos in which the effects of the radial movement of the image of objects are removed, aerial photography should be carried out with a minimum longitudinal and transverse overlap of images of 80% [27]. Additionally, the analysis [25] has shown that high-resolution aerial images allow manual visual identification of 0.3 mm cracks from a distance of 10 m to the surface, but it is achievable only in the case of sharp images and images with good exposure and the lowest possible image noise, while cracks with a width of 0.5 mm are clearly visible from a distance of approximately 7.5 m to building surface. In this phase, fieldwork equipment (UAV, total station, etc.) should be prepared as well.

2.2. Building Façade Survey

The second phase of AIM is conducted on site, referring to the data acquisition carried out through a survey of the building façade and includes two activities: (i) image dataset acquisition and (ii) survey of control points.

Image dataset acquisition is performed in accordance with flight parameters defined in phase 1. Before and during the flight, safety procedures should be conducted [28]. Prior to the flight, the UAV operator should: check if the UAV system is working properly; check the battery condition; ensure that all UAV equipment is properly secured; gather all necessary information for the safe performance of the planned flight; confirm that meteorological and other conditions in the flight area ensure its safe and efficient performance. During the flight, the UAV operator should: ensure that the flight does not endanger the life, health and property of people and does not disturb public order; use the UAV in a way that ensures compliance with the flight rules prescribed by current legislation; ensure that the flight is performed completely within the permitted area; ensure that UAV keeps a safe distance from obstacles; not be under the influence of alcohol or psychoactive substances, nor in such a psychophysical condition that prevents him from safely operating the drone.

Survey of control points is performed in order to increase the accuracy of the final orhofaçade image. The method of precise electronic tachymetry is applied for the survey. The process includes measuring the 3D coordinates of selected characteristic points on the façade of interest using a geodetic instrument—a total station. A minimum of three GCPs is required, while a minimum of five is recommended. Five to ten GCPs are usually enough, even for large projects [29], but increasing the number of GCPs will lead to higher accuracy of the final results [30].

2.3. Orthofaçade Generation

The third AIM phase includes data processing activities carried out in the photogrammetric software. The collected images and 3D coordinates of control points represent the input data for the tool, while the generated orthofaçade is the final output.

2.4. Crack Inspection

The final AIM phase refers to the crack inspection performed on the generated orthofaçade using the methods contained in the OpenCV library. The phase includes the following activities: (i) crack detection, (ii) determination of crack location and (iii) determination of geometrical parameters of crack.

The process of crack detection begins with the selection of a repeating rectangular façade element of interest at the generated orthofaçade. The selection is made with a simple tool that takes over the coordinates of the corners of the rectangle representing the observed block. It is envisaged that the selection of the observed block is reduced to the upper orthofaçade block, while all other blocks are vertically below. After that, based on the orthofaçade’s raster properties, the offset between the blocks in pixels is calculated and individual blocks are extracted. Each individual block is saved in a separate directory with a name that uniquely addresses the position of the block on the grid, so that in case of a detected crack, it is possible to reconstruct the precise location of the element and cracks. The number of files corresponds to the number of blocks viewed on the selected vertical. This concludes the observation of the façade as a whole and proceeds to the analysis of individual segments.

The next step is pixel segmentation in the selected block. Each individual file is accessed and a pixel analysis is performed taking into account a subset of adjacent pixels at the same time, calculating a threshold for that specific local region, and then performing segmentation. The aim is to separate the potential crack from the background, based on the difference in pixel intensity of each region. For the purpose of this action, the following was used: adaptive thresholding with OpenCV (cv2.adaptiveThreshold).

Thresholding types are:

1. 

cv.THRESH_BINARY

2. 

cv.THRESH_BINARY_INV

3. 

cv.THRESH_TRUNC

4. 

cv.THRESH_TOZERO

5. 

cv.THRESH_TOZERO_INV

Thresholding types are used according to OpenCV documentation:

void cv::adaptiveThreshold(cv::InputArray src, cv::OutputArray dst, double maxValue, int adaptiveMethod, int thresholdType, int blockSize, double C)

The AIM code applies an adaptive threshold to an array. The function transforms a grayscale image into a binary image according to the formula: **THRESH_BINARY**

The function parameters are:

• src—Source 8-bit single-channel image.
• dst—Destination image of the same size and the same type as src.
• maxValue—Non-zero value assigned to the pixels for which the condition is satisfied
• adaptiveMethod—Adaptive thresholding algorithm to use, see #AdaptiveThresholdTypes. The #BORDER_REPLICATE | #BORDER_ISOLATED is used to process boundaries.
• thresholdType—Thresholding type that must be either #THRESH_BINARY or #THRESH_BINARY_INV, see #ThresholdTypes.
• blockSize—Size of a pixel neighbourhood that is used to calculate a threshold value for the pixel: 3, 5, 7, and so on.
• C—Constant subtracted from the mean or weighted mean (see the details below). Normally, it is positive but may be zero or negative as well.

As a result of this activity, a new set of rasters is obtained that corresponds to the observed blocks. Within each individual raster, the potential crack is separated from the background based on the difference in the pixel intensity of the individual regions.

After the rasters were processed in this way, the function was applied in order to facilitate identification:

int cv::floodFill(cv::InputOutputArray image, cv::Point seedPoint, cv::Scalar newVal, cv::Rect *rect = (cv::Rect *)0, cv::Scalar loDiff = cv::Scalar(), cv::Scalar upDiff = cv::Scalar(), int flags = 4)

This feature uses and updates the mask and initializes the mask content. Based on OpenCV documentation, functions floodFill fill a connected component starting from the seed point with the specified colour.

The following step inverts every bit of an array:

void cv::bitwise_not(cv::InputArray src, cv::OutputArray dst, cv::InputArray mask = noArray())

The function cv::bitwise_not calculates per-element bit-wise inversion of the input array according to parameters:

• src—Input array.
• dst—Output array that has the same size and type as the input array.
• mask—Optional operation mask, 8-bit single-channel array, that specifies elements of the output array to be changed.

After applying the bitmask and inversion, a detection result is obtained and the crack is clearly identified.

The final step includes applying the mentioned simple tool that takes over the coordinates of individual points in order to determine the location, length, width and area of the detected crack. Crack information acquired with the AIM represents the basis for further steps in the process of building façade assessment.

3. Results

3.1. Preparation and Organisation of Inspection

The results of previous research [31], conducted on residential buildings built with industrial building technology in Novi Sad, in the second half of the XX century, have indicated the unsatisfactory technical condition of façade elements. After many years of exploitation and a lack of regular maintenance, there is a need to renew and improve façade performance, in order to comply with the requirements of current technical regulations and standards. Delaying renewal is not recommended, and some of the most important needs for improvement address the durability of the building envelopes.

In order to test and validate the proposed method, a case study was conducted. The case study focus was on the southeast façade of a high-rise (13-storey) residential building, built in 1972 in Novi Sad, Serbia (Figure 2).

UAV mission planning resulted in optimal flight parameters. Technical specifications of the Parrot ANAFI drone used for photogrammetric data acquisition are presented in

Table 1. Technical specifications of the Parrot ANAFI drone used in the case study [32].

UAV Specifications
Size unfolded	240 × 175 × 65 mm
Weight	320 g
Max. flight time	25 min
Operating temperature range	−10 °C to 40 °C
Max. horizontal speed	15 m/s
Max. vertical speed	4 m/s
Max. transmission range	4 km with controller
Max. wind resistance	50 km/h
Satellite Positioning Systems	GPS & GLONASS
Camera Specifications
Sensor format	6.194 × 4.646 mm
Sensor	1/2.4″ CMOS
Lens	FOV 180°
ISO range	100–3200
Image resolution	4608 × 3456 px
Focal length	4 mm
Diagonal crop factor	7.487

.

Due to the flexibility of the acquisition process and possible obstacles in front of the façade (external building staircase, power cables and trees), manual flight mode was selected. It was planned to perform flights with a camera oriented perpendicular to the façade, from a distance of about 6 m. Based on the defined flying distance and the UAV camera technical specifications, the GSD of the UAV images was calculated (2.1 mm). In the context of flight pattern, vertical flying paths were chosen for the survey. The details of the flight plan are given in

Table 2. Details of the flight plan.

Flight Parameter		Performance	Flight Pattern
Flight mode		Manual
Flying distance from the façade		6 m
Camera orientation		Perpendicular
GSD		2.1 mm
Area covered by a single image		9 m × 7 m
Image capture intervals		1 m × 1 m
Image overlap	Vertical	86%
	Horizontal	89%

.

3.2. Building Façade Survey

Image dataset acquisition was carried out during the winter months of 2021. Before the flight, UAV take-off positions were determined and marked on the terrain in front of the façade (V positions, Figure 3a) and at two associated building corners (D positions, Figure 3a). The façade was photographed from close range—from a distance of 6 m—and flights were controlled using the Parrot FreeFlight 6 mobile application. In order to ensure horizontal and vertical image overlap of 89% and 86%, respectively, the UAV flew along 18 vertical paths, capturing images at 1 m × 1 m intervals, with the camera oriented perpendicular to the façade surface. Two out of 18 paths (paths V1 and V18, Figure 3a) were positioned beyond the façade boundaries, in order to obtain accurate data on the edges of the façade surface.

The time required to conduct the mission was about 2 h and 1055 façade images were collected.

Survey of control points included measurement of 3D coordinates of selected characteristic details on the façade. Trimble S5 robotic total station was used to determine the positions of control points. A total of 12 points, evenly distributed at different height levels of the façade, were surveyed (Figure 3b).

3.3. Orthofaçade Generation

The processing of the collected data was carried out in the photogrammetric software Pix4D and resulted in a georeferenced point cloud, composed of over 35 million points (Figure 4a). The created point cloud was used to generate the orthofaçade with an average GSD of 2.2 mm (Figure 4b). The generated root mean square (RMS) errors of control points in all three coordinate axes were less than 1 cm. The total time required for the generation of the 3D point cloud and orthofaçade in Pix4D was approximately 3 h.

The achieved average GSD of the generated orthofaçade corresponds to the GSD of the initial photographs, determined in the flight planning phase.

The analysis of enlarged orthofaçades shows that most of the details on the façade are realistically and clearly presented (Figure 5). Thanks to well-planned data collection, this also applies to the edges of the façade surface, despite the fact that a smaller number of images are overlapped in these zones than in the central parts. Identified irregularities are mainly manifested in the form of negligible shadows on one side of the façade.

3.4. Crack Inspection

Within the crack inspection phase, the parapet façade elements of one vertical were analysed. From the orthofaçade image, 14 individual blocks were extracted and saved in a separate directory (Figure 6). The number of files corresponds to the number of blocks viewed on the selected vertical.

The following steps are performed by analysing individual segments, and the results of crack detection are shown in Figure 7.

After the extraction of façade elements, each individual element was accessed and a pixel analysis was performed, calculating a threshold and then performing segmentation. As a result, a new set of rasters was obtained for each observed block. On each single raster, the potential crack was separated from the background based on the difference in the pixel intensity of the individual regions (Figure 7b).

The final step of crack detection included bitmap inversion, which enabled a clear image of the crack (Figure 7c).

After crack detection, the coordinates of individual points could be used to determine information on the location of the detected crack in the selected coordinate system and the crack geometry.

4. Discussion

The efficiency, reliability, accuracy and objectivity of the automated building façade inspection process depend on the ability of the applied model to identify, locate and quantify damage on images acquired by the UAV. Most related studies have focused on crack detection only, without considering the geometric and spatial parameters of the detected crack, which represent key information for damage mapping and quantification of façade degradation.

The paper proposed the orthofaçade-based assisted inspection method—AIM; for crack detection and determination of location and geometrical parameters of detected crack. The AIM enables automatic crack detection, performed via computer code using open-source image-based methods. The model is defined by control points allowing the crack to be precisely located. By applying simple mathematical operations, it is possible to measure a detected crack based on the coordinates of its points. The method gives the user the ability to adjust the desired level of precision by defining a GSD that should not exceed half the dimension of the smallest object to be identified in the orthofaçade.

The data source used for façade crack inspection contributes to simplifying the façade assessment process. The AIM performs detection on a single high-quality orthomosaic image of the entire façade—an orthofaçade, generated from a large number of individual overlapping façade images collected with UAV from close range. Generated in an orthorectification-based process, performed by photogrammetric software, the resulting orthofaçade is geometrically corrected (“orthorectified”) and can be used to measure the true distances of features within the photograph. Additionally, by adding a façade control points obtained with the total station, the generated orthofaçade is georeferenced, which eliminates the need for traditional descriptive crack locating and allows highly precise identification of crack location. In other studies, the automatic detection of cracks from UAV images is mainly based on individual images or less often on one large, stitched façade image. Compared to the orthomosaic generation—applicable for large datasets, the image stitching method works well only if the area of interest is perfectly flat and therefore is recommended only for small datasets [17]. Other research [16] used the data from UAV sensors for crack location, but without control points, UAV images do not provide sufficient accuracy.

The proposed method was tested and validated through a case study conducted on a high-rise building façade. The presented results demonstrate that crack inspection using the AIM can be carried out efficiently, objectively and safely, while the reliability and accuracy of the final results are conditioned by the resolution and geometric accuracy of the orthofaçade generated within the process. The quality of the generated orthofaçade meets the requirements of the case study where the proposed methodology is validated. The applied AIM computer code successfully detected a crack in the orthofaçade image. The use of control points enables highly precise determination of the crack location, with accuracy on the sub-centimetre level and very accurate quantification of geometric parameters of the detected crack.

Contemporary building survey and condition assessment, science and practice demand a multidisciplinary approach, compiling and combining expert knowledge and decision making based on architecture, civil engineering, geodesy, material science and applied computer science. Despite all the efforts of the international scientific community to develop an automated building inspection framework, based on the combined application of different technologies for accurate data acquisition and reliable data analysis, introducing these technologies into the everyday practice of building façade assessment remains a challenge. This paper represents an effort towards the improvement of informed decision making regarding maintenance, repair, refurbishment or other actions during the exploitation stage of the built environment.

The AIM is suitable for the application on the surface of the façade elements that form an orthofaçade. The method can be applied to different types of façade cladding as long as the damage is manifested by a change in colour in the orthofaçade image.

Within future research directions, the established diagnostic model will enable the definition of the degradation function and the development of a prognostic model of the façade condition. Future AIM development is envisaged in the direction of machine learning and is conditioned by the expansion of the database.