HBIM for Conservation of Built Heritage

Abstract

Building information modeling (BIM) has recently become more popular in historical buildings as a method to rebuild their geometry and collect relevant information. Heritage BIM (HBIM), which combines high-level data about surface conditions, is a valuable tool for conservation decision-making. However, implementing BIM in heritage has its challenges because BIM libraries are designed for new constructions and are incapable of accommodating the morphological irregularities found in historical structures. This article discusses an architecture survey workflow that uses TLS, imagery, and deep learning algorithms to optimize HBIM for the conservation of the Nabatean built heritage. In addition to creating new resourceful Nabatean libraries with high details, the proposed approach enhanced HBIM by including two data outputs. The first dataset contained the TLS 3D dense mesh model, which was enhanced with high-quality textures extracted from independent imagery captured at the optimal time and location for accurate depictions of surface features. These images were also used to create true orthophotos using accurate and reliable 2.5D DSM derived from TLS, which eliminated all image distortion. The true orthophoto was then used in HBIM texturing to create a realistic decay map and combined with a deep learning algorithm to automatically detect and draw the outline of surface features and cracks in the BIM model, along with their statistical parameters. The use of deep learning on a structured 2D true orthophoto produced segmentation results in the metric units required for damage quantifications and helped overcome the limitations of using deep learning for 2D non-metric imagery, which typically uses pixels to measure crack widths and areas. The results show that the scanner and imagery integration allows for the efficient collection of data for informative HBIM models and provide stakeholders with an efficient tool for investigating and analyzing buildings to ensure proper conservation.

1. Introduction

The use of interactive parametric objects in the BIM platform allows for reliable modeling that provides information about a structure’s geometry and associated attributes. Even though the BIM method is primarily used during the design and construction phases of new buildings, it has grown in popularity in the heritage sector, known as heritage BIM. The Heritage Building Information Model (HBIM) can be used in heritage conservation, restoration, monitoring, management, and structural evaluation [1,2,3,4,5]. One of the current limitations of heritage BIM is the lack of parametric libraries designed for historical structures in BIM software [6]. This emphasizes the significance of creating new libraries with parametric architectural elements for various types of heritage structures that can be adapted as needed [7,8,9,10,11]. However, creating such resourceful libraries necessitates detailed and accurate surveys of the architectural elements [12]. The irregular and unique structures forming historic buildings make this task complicated compared to standard BIM modeling. Nowadays, it is possible to generate informative models for various types of objects, thanks to research activity in the field of 3D modeling using terrestrial laser scanners (TLSs) and digital photogrammetry. The surveyed data are then transformed into parametric modeling using a method known as scan-to-BIM using different BIM platforms [13,14].

TLS point clouds provide accurate 3D data coordinates for building surfaces quickly and in large quantities when compared to traditional surveying techniques [15]. It is critical in complex and large structure surveys to locate and optimize TLS stations to achieve complete registration of the entire scene [16]. Mobile mapping systems and SLAM (simultaneous location and mapping) techniques can enhance the scanning of complex heritage buildings [17]. The system can provide real-time 3D point clouds generated by automatic scan-to-scan registration with an accuracy of a few centimeters [18]. However, the generated points are a collection of discrete data records that lack semantic information about surface characteristics [19]. Additionally, the majority of TLSs can colorize their point clouds by acquiring digital images with a built-in camera attached. However, colored data may not be of sufficient quality for identifying structure surface features and their relevant conditions. The scanner’s position may not always coincide with the optimal camera position and time for color recording [20]. Several solutions are proposed for combining the advantages of laser scanners and photogrammetry to improve the color of laser point clouds [21]. The camera mathematical model is used to back project points from 3D object space to 2D image space coordinate systems, thereby attaching image information to geometric data. Additionally, cloud-to-cloud registration of photogrammetry and laser data improves recorded outcomes in heritage applications [22,23]. The common problem, known as the mixed pixel effect, which appears in the borders of cracks and the outlines of edges [24,25], can also present inaccurate data interpretation and surface crack drawing during HBIM tracing and modeling. Currently, photogrammetry is an efficient and cost-effective method of recording historical structures. Photographs taken at optimal standpoints and times provide an accurate representation of color and façade features. The processing pipeline in photogrammetric commercial software is standard and efficient for 3D reconstruction. Limitations in image quality, camera networks around the surveyed object, shadows, and model scale are the main issues, which could have an impact on photogrammetric processing and the final 3D point cloud [26,27].

The multisensory integration of TLSs and photogrammetry overcomes their respective bottlenecks, making it the most effective method for digitizing complex and large historical buildings and sites. Integration has the potential to improve geometric and radiometric data [28]. Several hybrid approaches have been proposed recently to improve scan-to-HBIM processing. For example, [5,29] discussed the benefits of photogrammetric UAV flexibility in inaccessible areas as a supplement to TLS occlusion. Other researchers used multisensory data to improve the quality and quantity of input data at scan-to-BIM, improving modeling results [30,31]. Ref. [32] used point clouds, images, and historical data to accurately model complex and uneven surfaces of historic buildings in the BIM platform. The ability to incorporate high-level structural information into the HBIM model to highlight surface pathologies such as material degradation, environmental conditions, and crack damage is critical for conservation activities [33,34,35]. Several studies investigated the mapping of material components and decay in a BIM environment using a color legend with the Revit Materials Browser [36,37]. Others kept images or orthophotos alongside BIM surfaces as complementary and attached data to help with heritage conservation [38,39]. However, in heritage applications, employing BIM models that accurately represent the building in its original state remains a challenge [40].

On the other hand, the process of recording and measuring cracks through human-based visual inspection is subjective and time-consuming [41]. Quantifying and monitoring the extent of damage and decay using metric surveys, gypsum strips, and sensors installed in historical structures requires professionals, and there are difficulties entailed in distributing the devices in large-scale structures [42,43,44,45]. These methods can only provide discrete point measurements at specific locations, and some of them require long-term bonding with the structure surface, making them unsuitable for aesthetic and functional reasons in heritage applications. Recently, machine/deep learning models have been changing the way researchers detect, interpret, recognize, and classify historical building components [46,47,48,49]. Algorithms based on artificial neural networks employ multiple processing layers to learn feature representations of data at various levels of abstraction. Models can capture contextual information by considering larger spatial contexts. This enables them to understand and distinguish edges based on surrounding patterns, leading to more robust results, such as the calculation of area and perimeter for a certain feature. The algorithms used to detect cracks in 2D photos typically use pixels to measure crack width and area. However, the photographs are distorted due to the camera’s central perspective projection and suffer from a lack of uniform scale, making it difficult to determine the dimensions within the photograph [50]. Thus, different methods have been proposed for converting the pixel unit system to the metric unit system by utilizing the camera interior parameters [51,52] or the laser calibration method [53]. However, limitations such as image perspective, distance, and angle of camera optical axes with respect to the surface can impact the results and are thus still being discussed. Contrary to 2D algorithms, deep learning processing in the 3D unstructured 3D point cloud is very challenging. The main challenge is in the complexity and variety of point clouds, which results from irregular sampling, variable object density, various object types, and the availability of labeled datasets [54]. Fully automated feature extraction methods from point clouds are currently at the cutting edge [55,56]. On the other hand, orthophotos eliminate perspective distortions in imagery and provide uniformly scaled imagery, making them a useful tool for quantifying and positioning surface features [57]. An orthophoto based on accurate and reliable 2.5D DSM that removes all imagery distortion is also called a true orthophoto [58,59]. Orthophotos generated by photogrammetric processing are widely used in heritage applications [38,50,60,61]. Their quality is affected by several factors, including image resolution, camera parameters, and the accuracy of the digital surface model (DSM). However, using the photogrammetric pipeline to generate DSMs from images has several challenges, including texture-less elements, error propagation, and noise from long image sequences of large-scale structures, all of which may result in matching failures that affect the SFM results used to generate the DSM [27,62].

This paper proposes a multi-source documentation approach, depicted in Figure 1, to enrich the HBIM platform with high-level data about surface conditions that are useful for conservation work. In addition to developing new resourceful Nabatean libraries, the proposed approach improved HBIM by adding two data outputs. The first dataset was the TLS 3D dense mesh model, which was enhanced with high-quality textures extracted from independent imagery. The produced models were combined with the parametric model to provide a realistic representation of the decay regions. The second dataset used true orthophotos produced by TLS and independent images taken at the best time and position for proper radiometric information for the realistic texturing of the HBIM model, and this was combined with a deep learning algorithm to automatically detect and draw the outline of surface features and cracks on the HBIM model, along with their relevant statistical parameters. The use of deep learning for true ortho images provided the practical metric information needed for damage quantifications and helped overcome the previously discussed issues and limitations of using deep learning in 2D non-metric imagery or 3D point clouds. During true orthoimage production, a reliable digital surface model (DSM) derived from TLS’s point cloud was used to remove perspective distortions from independent perspective photos. This resulted in an accurate metric system. The extracted feature components, along with their statistical attributes like length, area, and perimeter, were then attached to the BIM model for analysis of all types of surface weathering forms, including their extent, typology, cause, and the conservation activities required. Data fusion was used during the recording of the AlDeir monument in the ancient city of Petra of Jordan, a UNESCO World Heritage Site. To summarize, the following contributions are covered in this paper:

• Creating a new HBIM library of Nabatean-built architectural elements with detailed parametric objects representing the as-built condition at the level of development (LOD) 300.
• A fusion-based approach using TLS and high-resolution imagery survey results to enrich scan-to-BIM with realistic renderings of surface material decay.
• A deep learning approach using true orthophotos to automate the detection and quantification of façade degradation and cracks for enriching the HBIM.

The following sections comprise the paper’s structure: Section 2 covers data collection and pre-processing. The creation of the Nabatean BIM library is discussed in Section 3. Section 4 describes how diagnostics data were implemented in the HBIM. The application of deep learning in HBIM is discussed in Section 5. Section 6 discusses the results and Section 7 presents the conclusions.

2. Data Collection

2.1. AlDeir Monument in Petra City

The ancient city Petra, located in southern Jordan, was the capital of the Nabataean empire from 400 B.C. to 106 A.D. Because of its location close to the incense trade routes, the Nabataeans, a group of Arab nomads, invested in making Petra a significant regional trading center [63]. Many fascinating ancient world monuments of exceptional architectural quality can be found in Petra. Petra’s temples, tombs, theaters, and other structures occupy more than one thousand square kilometers. The structures were carved into the rose-colored sandstone cliffs. In 1985, UNESCO designated Petra as a World Heritage Site. It was chosen as one of the world’s New Seven Wonders in 2007 [64]. Figure 2a depicts the AlDeir monument (monastery), one of Petra’s largest monuments. According to [65,66], the AlDeir monument, which is located northwest of the ancient city, is thought to have been carved in the middle of the first century AD. More than eight hundred steps cut into the rock lead up to a 47 m wide by 48.3 m high, remarkably well-preserved façade. The upper part of AlDeir’s façade is made up of two half-pediments that frame a circular tholos form with a conic roof and an urn, similar to the Al-Khazneh (treasury) model shown in Figure 2b. The tholos is framed by columns with Nabataean “horned capitals” and pediments are linked by a continuous frieze that has a simple design of alternating triglyphs and disks. The facade is structured by a doorway architecture with pilasters, capitals, and other decorative elements. During the Christian Byzantine era, it was used as a monastery, with the interior hall repurposed as a chapel and crosses carved into the back wall. Worshippers and priests congregated in the open area in front of this temple, making it a popular pilgrimage site. Recent environmental monitoring of Petra monuments has revealed significant recession and weathering processes. The World Monument Fund inscribed Petra on its list of the world’s one hundred most endangered monument assemblies in 1998. This can be related to the nature of the building materials used at these sites, which are primarily porous inorganic materials, as well as the uncontrolled environmental conditions surrounding them, such as salt damage, which is the primary damage process for Petra monuments [67,68]. This highlights the importance of developing a management and conservation strategy for this important heritage of rock-cut architecture.

2.2. Sensor Applied

In our research, the surface 3D point cloud was collected using the Mensi GS100 Laser Scanner. This scanner provides a panoramic view with a 360-degree horizontal and 60-degree vertical field of view. The device is capable of measuring up to 5000 points per second with a precision of 6 mm at a maximum capture range of 100 m [69]. During data collection, the scanner’s attached camera takes a calibrated video snapshot with a resolution of 768 × 576 pixels, which is then automatically mapped to the corresponding point measurements. A single laser scan cannot typically cover an entire structure. The amount of self-occlusion and obstacles, as well as the object’s size in relation to the sensor range, all affect how many scans are necessary. To resolve the occlusions, five different viewpoint scans were performed during the data collection at the AlDeir monument, which resulted in nearly 6 million points with an average ground sampling distance (GSD) of 3 cm. After collecting TLS data, noise data were removed in the first step of processing. Cloud Compare software v2.11.3 was then used to align all of the scans into a local coordinate system without georeferencing, as the research’s goal was to create a BIM model. Co-registration was achieved by using corresponding tie points rather than Ground Control Points (GCPs). Figure 3a depicts one of the collected scans, whereas Figure 3b depicts the registration results of the different scans. According to the registration data report, the alignment error (RMS error) was around 30 mm. Figure 4 depicts AlDeir’s final model with shaded and colored data. The coloring was provided from the camera attached to a TLS. The large and complex building may have had different lighting conditions due to multiple scans taken at different times, resulting in a non-homogeneous appearance and color jumping in the final model. Additional digital images were taken with a Nikon D2x camera at a resolution of 4288 × 2848. These images were taken near-simultaneously to ensure consistent lighting conditions and radiometric properties for laser data coloring. This improved the visual quality of surface features and orthophoto production, which will be discussed in the following sections.

3. Nabatean BIM Library

HBIM is a novel way to model historic structures with remotely sensed data. HBIM is a unique collection of reusable parametric objects based on architecture survey data. Parametric objects capture all information about the geometry of the elements and have customizable parameters linked with them. Once the parametric objects have been modeled, libraries of the modeled elements should be generated and used as a BIM plug-in to make reconstruction, rehabilitation, management, and maintenance processes of architectural heritage more efficient for the rest of its life cycle. Although BIM software has a standard objects library, the uniqueness of heritage building elements poses an important challenge when modeling their geometry. Creating new detailed parametric objects necessitates precise survey data collection and the determination of the Level of Detail (LOD). The LOD determines the accuracy of an element’s geometry and associated information [70,71]. The American Institute of Architects (AIA) defined five Levels of Development (LODs) to specify the BIM model’s detailing levels. LOD 100 contains the least amount of graphical and embedded information and LOD 400 contains the most of both [72]. Our research aimed to achieve LOD 300, which means that the BIM model would include both the built geometry and the condition data required for management and maintenance using both laser data and independent imagery. In the framework of HBIM, the point clouds are exported to the BIM platform after they have been registered. While the used BIM platforms have a large family library, few predefined objects met the needs of modeling our project, necessitating the creation of new Nabataean architectural element families. These families simplified the process of rebuilding 3D BIM models for other Nabatean buildings with similar shapes. The Autodesk Revit platform was used for BIM modeling because it is a versatile and powerful modeling tool that allowed us to create modeled families directly for the project. There were two phases identified by the AlDeir building’s BIM modeling. The first phase involved modeling regular architectural features like plain walls, cylindrical and rectangular columns, and exterior walls. The second phase involved creating new libraries of common heritage features of the Nabataean built heritage.

Nabataean building facades share many architectural and ornamental features. The AlDeir building’s HBIM documentation, which included well-preserved typical Nabatean architectural elements, enabled the creation of 3D parametric models that could be used as a valuable library of Nabatean built heritage. It will set a precedent for future documentation of the built heritage in the ancient city of Petra. Ad Deir’s architectural design is a sophisticated blend of Nabatean and Hellenistic elements, as well as an evolution of the Nabataean style. The primary architectural elements are pediments, column capitals, tholos, cornices, continuous friezes, and door frames. The tholos is framed by columns with Nabataean “horned capitals” and the pediments are joined by a continuous frieze with alternating triglyphs and disks. The facade is constructed by a doorway architecture with pilasters, capitals, and other decorative elements.

The modeling strategy for building components was adapted according to the element’s nature. In a BIM workflow, the existing 2D and 3D families make modeling simpler and faster. However, many structural elements found in Nabataean monuments are unique and do not have equivalents in existing BIM libraries. As a result, using Autodesk Revit, the monument was modeled with both system and in-place families. The segmentation of the object’s point cloud served as the primary starting point for the scan-to-BIM modeling process. Different horizontal and vertical cross-sections were generated from the AlDeir point clouds to identify and quantify the various architectural elements in the monument, as depicted in Figure 5a. The system families built key elements such as walls, roofs, and floors. In system families, the dimensions of the walls and roofs and door width, length, and depth were crucial for creating the correct profiles and demonstrating the feature parametric function, which could modify each element. In addition, direct tape measurements were used to determine the thickness of some components where point clouds were missing due to occlusion.

The lack of libraries that satisfied the HBIM project requirements of modeling the unique cultural elements and decorative components of the Nabataean monuments necessitated the full modeling of these families. In our experimentation, this problem was resolved by creating in-place components using simple Boolean operations. Revit is more effective with available 2D and 3D families than using model-in-place elements, which can increase file size and reduce software performance. However, in-place families were more adaptable to model unique, complex, and free-form objects within the project context. In-place elements provided the flexibility to make precise adjustments and modifications to the model directly in response to specific project requirements and the unique characteristics of each architectural element. New in-place families were developed and implemented in the BIM platform and the profiles were extracted and drawn from point cloud data. Different family parameters were defined, such as the dimensions, materials, and text that described the state conditions. To locate the outline of these components, the point cloud was divided into sections and elevations. The components were then traced and modeled, as shown in Figure 5b,c. The entablature, three-quarter and half columns with Nabataean capitals, tholos elements, cavetto cornice, and various decorative elements were created using Revit’s in-place regular and free-form modeling tools, depicted separately in Figure 6. The elements were classified into the appropriate category within the project as a family created for this purpose. The database and the new library were linked together to provide a practical method for modeling common Nabataean architectural features. Figure 7 depicts the complete AlDeir BIM model, which included all architectural elements.

4. Implementing Diagnostics Data in HBIM

The use of HBIM and informative data to record building deterioration and threats improves structural health quality control. The proposed data fusion approach, which incorporated TLS, imagery, and computer vision, enhanced the HBIM by adding two data outputs that reflected structure conditions. The first dataset contained a TLS three-dimensional dense mesh model that was enhanced with high-quality textures derived from various images. The second dataset used a true orthophoto from TLS and imagery data for texturing the BIM model.

4.1. Realistic Renderings of Surface Material Decay

Texture mapping is the visual and graphic representation of the mechanical and physical properties of materials. The current BIM platform’s primary goal was to manage the geometry of the new structures, so texture was not as important. The majority of HBIM applications handle texture by either creating a custom texture from the object image or using the program’s default texture library [31,73,74]. However, neither provides consistent and reliable visual effects of the as-built surface. Surface mesh, on the other hand, enables the accurate rendering of irregular architectural shapes and as-built conditions. The meshing process converted point cloud data into triangle polygon representations. Although the laser scanner had an attached camera that collected color for each 3D point cloud, the color data could be of insufficient quality to display surface conditions and features, as shown in Figure 8. The ideal camera position did not always coincide with the scanner position. The suggested method combined TLS geometric data with independent photos. These photos were taken at the optimal positions and times, considering the shadow and occlusion issues, to provide realistic representations of surface color. Before utilizing the complementary characteristics of both datasets, the selected image needed to be registered with TLS. In our approach, conjugate control points were defined manually from the selected image and registered TLS point cloud. This method solved the camera’s external parameters (orientation and position). The solved parameters were then used to find image texture coordinates and project RGB values for the laser scanner point, using the collinearity Equations (1) and (2). Figure 9 illustrates how these equations were obtained from the central projection of an object point via the sensor plane to the perspective center (PC) of the camera. The angular parameters (ω, Ø, κ) were used to extract the camera rotation matrix elements rij in the equations. In the equations, the focal length of the camera is c, the camera position coordinates in the object space are (X_o, Y_o, Z_o), and the principal point coordinates of the image are (x_o, y_o). The object point coordinates are (X_A, Y_A, Z_A), while the image point coordinates are (x_a, y_a). We used our own C++ code to map the texture RGB color values of the image to 3D point clouds. Figure 10b depicts the colored point cloud, which was then meshed in obj format files, as shown in Figure 10c,d, and managed in the REVIT platform. This HBIM model allowed us to access various models in real time, including point clouds, textured mesh models, and parametric elements, resulting in a powerful tool that benefits all stakeholders.

$$x_a=x_0-c\frac{r_{11}\left(X_A-X_o\right)+r_{21}\left(Y_A-Y_o\right)+r_{31}\left(Z_A-Z_o\right)}{r_{13}\left(X_A-X_o\right)+r_{23}\left(Y_A-Y_o\right)+r_{33}\left(Z_A-Z_o\right)}$$

(1)

$$y_a=y_0-c\frac{r_{12}\left(X_A-X_o\right)+r_{22}\left(Y_A-Y_o\right)+r_{32}\left(Z_A-Z_o\right)}{r_{13}\left(X_A-X_o\right)+r_{23}\left(Y_A-Y_o\right)+r_{33}\left(Z_A-Z_o\right)}$$

(2)

4.2. HBIM Texture Mapping

True orthophotos eliminate all imagery distortion by employing a digital surface model (DSM), a 2.5D representation of the structure, yielding uniformly scaled imagery that allows the user to position objects. In our approach to producing true orthophotos, TLS point cloud was used for developing a reliable DSM, while radiometric information was taken from independent and oriented images obtained from the photogrammetric process. The process began with sampling the laser scanner point cloud into a regular grid DSM plan, with each cell (pixel) of the plan containing a unique value associated with the model point, as shown in Figure 11. Another grid plan with the same DSM pixel size was created to serve as a framework for true orthophotography. The relevant imagery data were then projected into the orthophoto pixel using DSM information and camera parameters using collinearity equations. The RGB value of the true orthophoto was calculated using pixels from the source image corresponding to the visible points. The white labels were applied to invalid pixels or pixels from shadow areas (which lacked DSM data). We used C++ as a programming language. Figure 12 shows the DSM and the resulting true orthophotos at a 0.6 mm/pixel resolution for the monument facade.

Users can detect surface decay and object deterioration in heritage structures using texture mapping with true orthophotos at a uniform scale. Using HBIM texture data, we could also reconstruct realistic renderings of the built heritage. Because the generated orthophotos had the same dimensions as the corresponding HBIM surfaces, they were precisely and consistently warped. The true orthophoto file could be accessed and placed directly on the decal for texturing the objects, as shown in Figure 13, because its width and height corresponded to the actual size of the decal on the software options bar.

5. Deep Learning in HBIM

Computer vision-based edge detection technology has been extensively researched for heritage buildings. Several techniques have been investigated, including threshold segmentation [75,76], morphology [77], wavelet transform [78], and filter-based algorithms [76,79]. Traditional computer vision algorithms typically focus on analyzing specific features within images, using parameters designed for specific datasets. To address this limitation, researchers have increasingly turned to machine learning techniques for image-processing tasks [80,81]. Since convolutional neural networks (CNNs) were presented by Lecun et al. in 1998, they have been applied to various tasks including image classification [82,83], object detection [84,85], semantic segmentation [86,87], and crack detection [88,89].

Deep learning models, particularly those for semantic segmentation, have been extensively studied and have shown promising results in edge detection. Deep learning models can extract edges from input images without the need for explicit feature engineering [90]. The models can capture contextual information by considering larger spatial contexts. This enables them to understand and distinguish edges based on surrounding patterns, leading to more robust results, such as the calculation of area and perimeter for a certain feature [91]. Research on edge detection based on deep learning has proposed techniques such as image classification, object recognition, semantic segmentation, and the classification of buildings’ degradation state [92]. In recent studies on crack detection and segmentation, various deep learning architectures have been applied [93,94].

Deep learning techniques require enough data to complete the model training phase. However, the current dataset comprising images from heritage sites fell short of meeting these requirements. To address this particular problem, researchers have turned to transfer learning as an alternative approach [95,96]. Transfer learning, a powerful technique in deep learning, allows models to leverage knowledge gained from one task and apply it to another related task. By utilizing pretrained models trained on extensive datasets and complex tasks, transfer learning enables the initialization of new models for solving novel problems. This approach circumvents the need to train a model from scratch, significantly reducing the demand for training data and computational resources. In recent studies on image analysis and segmentation, transfer learning has been employed as an effective approach. Xu et al. [97] used the transfer learning approach to automatically segment and label histopathology images. Zhang et al. [98] applied transfer learning to identifying and extracting crack information from images of earthen heritage sites.

5.1. Holistically Nested Edge Detection (HED)

Holistically nested edge detection (HED), a CNN algorithm, employs a distinctive approach for multi-scale image convolution, based on the VGG16 architecture with modifications. In this model, the fully connected layers are removed after the last pooling layers, retaining only the convolutional layers, as shown in Figure 14. HED progressively reduces image resolution through five sets of 3 × 3 convolutional layers separated by 2 × 2 max pooling layers. After each set, a side output layer is generated. The final output, predicting pixel edge likelihood, is obtained by optimally fusing these five side output layers. Prior to HED application, images undergo preprocessing, including cropping, normalization, and blob construction.

The training process for HED involves utilizing a labeled dataset to optimize the model’s parameters and enable accurate edge detection. The training procedure first includes dataset preparation, where a dataset containing images with corresponding edge annotations is curated. Second, images from the dataset are preprocessed and fed into the HED model as input. The preprocessing may include normalization, resizing, or any other necessary transformations to ensure uniformity. Third, the model is initialized with its architecture, weights, loss function definition, and optimization algorithm. Then, the model is trained through multiple iterations (epochs) over the entire dataset. Finally, the model’s performance is evaluated on a separate validation set to ensure that it generalizes well to unseen data and does not overfit the training set. The model is implemented using the publicly available Caffe library; this Caffe model is encoded in two files: a text Caffe JSON file with the model definition and a text file with the neural network weights. For a detailed overview of the HED architecture and its functionality, refer to Xie and Tu, Kokkinos [99,100].

5.2. Deep Learning Processing

In our approach, a deep learning algorithm was used to process true orthophotos produced from TLS and imager data fusion. True orthoimages offer numerous advantages, including metric accuracy and radiometric information that can be used to quantitatively and qualitatively analyze various datasets. The use of deep learning for structured 2D true orthophotos produced segmentation results in metric units and helped overcome the previously discussed limitations of using non-metric photos and 3D point clouds.

Prior to implementing the proposed methodology in the case study, an initial experiment was conducted to assess the effectiveness of the algorithm for edge detection, feature extraction, and image segmentation under different conditions. The results were compared with those obtained using traditional methods such as the Canny and Sobel operators, as illustrated in Figure 15. The experiment involved the use of various examples of historical building facades located in Jordan. For instance, example (A) represents the facade of Al-Abidit palace in Jordan; the palace was constructed in the second century BC by Hyrcanus during the reign of Seleucids IV and has large stones and different types of cracks. This example also demonstrates variations in tone for the same feature from one location to another in the image. Example (B) depicts the left door of the Alkhazneh monument in the ancient city of Petra. This example was chosen due to the significant weathering and numerous forms of erosion visible on the walls, as well as the low lighting conditions. In example (C), a part of the main facade of Amra palace in Jordan is represented, a UNESCO World Heritage site built in the Jordan desert between 705 and 715 A.D during the reign of the Umayyad Caliph Walid I. This example exhibits irregular stones with varying sizes and colors, as well as irregularities in architectural features. These examples were captured at different times and under diverse conditions. The selection of these images aimed to ensure diversity, enhance contrast, and present challenges for automated segmentation. The implementation of the HED model was performed using Python 3.9. Figure 15 illustrates the effectiveness of traditional edge detection algorithms such as Canny and Sobel in extracting edges.

In contrast, the proposed pretrained network algorithm, HED, enhanced crack detection capability and provided labeled contextual semantic information about edges and cracks, including area and length. This information can be utilized for material conservation analysis. Therefore, employing deep learning to offer informative and semantic segmentation in the digital cultural heritage domain can aid in identifying various architectural elements and surface details with precision, thereby enhancing the process of Historical Building Information Modeling (HBIM) for historical buildings. The comprehensive process of employing the HED model for edge detection and feature extraction and then image segmentation of the main facade’s orthophoto of the AlDeir monument is illustrated in Figure 16. It can be observed that, at each step, a side output was extracted based on the scale. Five different scales were used to extract the image features. Following resizing to the original scale, the connected component step was executed, considering the features. OpenCV’s Gaussian blur and Otsu’s threshold were then employed to connect components. All segments were labeled, and statistical analysis was performed for each labeled segment. The analysis included calculations for area, equivalent diameter as the diameter of the circle whose area was equal to the area of the segment, perimeter, mean intensity, and solidity as the area of a segment divided by its convex hull area. The next step was to filter very small objects that did not represent a visible feature in the image.

Table 1. The statistical results of some selected labeled segments representing cracks and decay in the AlDeir facade.

Label	Area (m2)	Equivalent Diameter (m)	Perimeter (m)	Mean_Intensity in Red Band	Mean_Intensity in Green Band	Mean_Intensity in Blue Band	Solidity
13	0.32	0.63	1.91	38.3	73.0	135.2	0.91
18	0.05	0.24	0.42	63.7	107.3	164.3	0.98
31	0.14	0.41	0.93	15.7	45.3	100.6	0.91
50	0.54	0.83	2.60	13.8	53.5	126.5	0.92
78	0.21	0.52	1.40	77.1	116.4	178.3	0.88

displays the statistical results for some labeled segments that identified cracks and decayed areas on the main facades of the AlDeir monument. The whole vector data derived from output segmentation, as well as the Excel sheet containing statistical information, were integrated into the Historic Building Information Model (HBIM). They will be repetitively produced over time to enhance the model for conservation and monitoring purposes.

6. Discussion

HBIM and BIM differ fundamentally in terms of information content and modeling approach. HBIM emphasizes the importance of understanding the conservation status of structure and materials, providing stakeholders with a knowledge tool to aid in diagnostic investigations. Recent scan-to-HBIM approach challenges include issues with modeling irregular features, a lack of pre-defined libraries of parametric objects, and mapping the decay state of built heritage, required for structure assessment. The proposed workflow presents a novel study that evaluated the potential advantages of combining TLS, imagery data, and image processing to enhance the HBIM platform with shared and useful data for conservation work. In addition to creating new HBIM libraries of Nabatean built heritage, the proposed approach introduces a new perspective on HBIM modeling focused on the automation of damage and crack detection and quantification using deep learning. The fusion method is adaptable because images can be taken independently of the TLS camera at the optimal time and location for radiometric data collection. This paper’s major contributions to cutting-edge approaches are as follows:

1. This work provides a complete set of “as-built” parametric models to aid Nabatean design and preservation activities. This project developed a new Nabatean-built HBIM library as a plug-in Autodesk Revit family with detailed parametric objects that can be adapted as needed.

2. Texture mapping is also useful in cultural heritage to demonstrate structural conditions, material degradation, and stages of rebuilding. The majority of HBIM apps manage texture by either developing a customized texture collected from the object pictures or using the program’s default texture [31,35]. Many studies have been conducted on the use of color legends to map material deterioration and surface pathologies in BIM models [36,37,101]. The results are unrealistic visual renderings of structure decay states that need conservation. The workflow combined TLS data with high-resolution imagery obtained independently at the optimal position and time for true orthophoto production, which was then utilized to texture the final HBIM model. A reliable DSM from TLS was used to remove perspective distortions from the photos during true orthoimage production, resulting in metric unit data that could be used directly for quantitative analysis. As shown in Figure 13, the orthophoto and accompanying HBIM surfaces had similar proportions, allowing for precise and consistent texture warping across the BIM geometry. The findings, as depicted in Figure 17a,b, provide a clear interpretation of surface features and serve as a useful tool for accurately representing the types, nature, and spatial extent of façade degradation.

3. Modeling historic buildings with their structural deformations and cracks is still a challenge in HBIM applications. Pathologies can be manually modeled and vectorized in HBIM [30,102] or with virtual gauges [73]. Manual work may be subjective and time-consuming. Machine learning and computer vision algorithms have shown excellent results in detecting cracks in 2D photos. However, due to the camera’s central perspective projection, such photographs lack a uniform scale, making pixel-based dimension estimation difficult. Several methods were proposed to convert these pixels to the metric unit system by using the camera parameters [51,52]. Even so, issues concerning image perspective, camera distance, and the angle of the camera optical axes in relation to the surface may still have an impact on the outcome. In the present work, the authors investigated the potential of deep learning approaches for the supervised classification of irregular heritage data using true orthophotos. The true orthophotos preserved the planimetric dimension of the geometry in the 2D space, which simplified data manipulation and helped overcome issues with using deep learning in non-metric or 3D point cloud processing. Figure 17c demonstrates the rich extraction efficiency of the suggested pre-trained network algorithm, which enhanced the ability to detect cracks and provided labeled and contextual semantic information about edges and cracks, including area, length, and perimeter, which can be used for material conservation analysis. The vector data derived from output segmentation were subsequently mapped in the BIM model, as seen in Figure 18. Contextual and statistical information, as well as other derived data, were linked to the Historic Building Information Model (HBIM), as shown in Figure 19, to provide a single platform for all stakeholders involved in the conservation process.

7. Conclusions

This paper described a multi-source workflow for an informative HBIM platform aimed at improving data sharing and collaborative work among all specialists, particularly those involved in monitoring and conservation interventions. TLS and imagery surveys were used to create a resourceful library for Nabatean architectural elements and develop a non-invasive analysis to quantify structure damage levels. The proposed approach enriched the HBIM with two data outputs to evaluate and map degradation in the façades. The first dataset was the TLS 3D dense mesh model, which was enhanced with high-quality textures extracted from independent imagery. The second dataset utilized true orthophotos and a deep learning algorithm for HBIM texturing and to automatically detect and draw the outlines of surface features and cracks, with their relevant statistical parameters. A reliable DSM model derived from the TLS point cloud was used to remove perspective distortions from the photos during true orthoimage production, resulting in metric unit data that could be used for quantitative analysis. These findings could be used to guide conservation efforts and develop adaptable and effective monitoring programs that assess the extent of damage and predict a defect’s potential growth.