3D Data Fusion for Historical Analyses of Heritage Buildings Using Thermal Images: The Palacio de Colomina as a Case Study

Abstract

In the framework of built heritage monitoring techniques, a prominent position is occupied by thermography, which represents an efficient and non-invasive solution for these kinds of investigations, allowing the identification of phenomena detectable only in the non-visible range of the electromagnetic spectrum. This is of extreme interest, especially considering the possibility of integrating the radiometric information with the 3D models achievable from laser scanning or photogrammetric techniques, characterised by a high spatial resolution. This paper aims to illustrate how combining different geomatics techniques (in particular, by merging thermal images, laser scanning point clouds, and traditional visible colour photogrammetric data) can efficiently support historical analyses for studying heritage buildings. Additionally, a strategy for generating HBIM models starting from the integration of 3D thermal investigations and historical sources is proposed, concerning both the multi-temporal modification of the volumes of the building and the individual architectural elements. The case study analysed for the current research was the Palacio de Colomina in Valencia, Spain, a noble palace—now the headquarters of a university—that, during the last few centuries, has been subjected to considerable transformations in terms of rehabilitation works and modification of its volume.

1. Introduction

In the last few years, we have witnessed the increasing use of many different sensors for the documentation and analysis of built cultural heritage, ranging from measurement tools over multiband and multispectral imaging, thermography and many more [1,2,3]. The integration of different sensors, techniques and 3D sensing strategies has become a standard for multi-scale and multi-source documentation processes [4]. Currently, true scale documentation of the status quo is typically carried out using laser scanning [5] or photogrammetry by means of digital cameras, following a terrestrial approach (close-range photogrammetry) [6] or using UAV (unmanned aerial vehicle) systems, that in recent years have become increasingly popular in the framework of built assets documentation [7]. Both techniques can provide accurate and high-resolution point clouds, giving the basic data for as-built CAD (computer-aided design) plans, or might be used in a scan-to-BIM workflow to create a digital derivate/digital twin in the form of a historic building information model (HBIM) [8]. In this way, visible details captured by the point cloud, proved and extended by information of extensive on-site observations, are mainly considered. However, for an adequate investigation of the history of a building with several constructive changes over time, a combination of different sensor techniques, historical sources and other scientific disciplines is mandatory [9,10,11].

Among others, infrared thermography (IRT) represents a powerful tool in the framework of building physics and in the field of cultural heritage monitoring [12,13,14,15], where it is mainly used for the non-destructive and contactless observations of occluded construction details and the detection of damage underneath the object surface. Anna Luib [16] gives a recent overview with basic information, many sample applications, best-practice guidelines and an overview of European institutes involved in research focused on the use of IRT for built heritage.

However, the biggest problem of thermal imaging is still the lack of sufficient sensor resolutions to capture bigger areas as a whole. Resolutions of around 1 to 3 megapixels only allow a low-spatial overall observation, or single detail images are needed to focus on small areas. A combination of several thermal images can solve this problem, as well as the integration of IRT imagery and 3D metric data characterised by a higher spatial resolution [17,18,19]. Different ways to process the data are as follows. Adamopoulos and Rinaudo gave an overview of different ways of close-range sensing data fusion [3]. Adam Dlesk et al. [20] presented a photogrammetric combination using a thermal camera with an integrated RGB camera. The estimated internal orientations of the two cameras were calculated and one of the RGB bands was replaced by the monochrome thermal image before aligning the images. Patrucco et al. [21,22] presented an SfM-based (structure-from-motion) photogrammetric workflow to map oriented thermal images on 3D surface models generated by high-resolution visible colour images acquired from a UAV system.

In this research study, two different workflows for the combination of thermal images with photogrammetry will be further developed, and a data fusion strategy with laser scanning and visible images will be presented, giving rise to the possibility of combining high-resolution surface measurements with a bundle of thermal images. In this preliminary part of the research, one of the main purposes is the evaluation of the effectiveness of both workflows, in connection with the specificities of the proposed case study, the flexibility of the two approaches, the manual involvement of the operator and the quality of the final results. The first strategy is based on processing a block of TIR images following an SfM-based photogrammetric pipeline, to orient a thermal dataset in the same reference system of LiDAR (light detection and ranging) scans and oriented visible images, exploiting the higher spatial resolution of these kinds of data to perform an effective data fusion between geometry and the information connected to the radiometry. The second strategy is based on the projection of a thermal texture—derived from a mosaicking procedure—on a dense and accurate 3D model derived from a LiDAR point cloud.

By combining several thermal images to represent a single façade or even the entire building, the lack of resolution will be compensated, and the level of thermal details will increase. Hereby the radiometric data will be mapped on true scale 3D measurements allowing a correct scaling and location of all information. Furthermore, the thermal 3D model and thermal orthoimages will be transferred into an HBIM, following a scan-to-BIM workflow [22,23,24,25]. Within the HBIM, the data fusion of geometry, visible colour, radiometry and historical sources will be processed. Combining all this information metrically correct in one information system may help heritage experts in the analysis and decision-making processes. Specifically, one of the aims of the current research is to demonstrate how this combination of heterogeneous—geometric, radiometric, historical, etc.—data in a reality-based HBIM can efficiently provide valid support to historical analyses.

2. Materials and Methods

In the subsequent sections, the image-based and range-based methodologies will be presented. Therefore, the acquisition of the primary data, including the point clouds derived from the terrestrial laser scanning system, the digital visible colour images, the thermal infrared (TIR) images, and also the processing of the data, will be shown.

2.1. Case Study

The Palacio de Colomina (Colomina Palace) served as a case study for this research. It is located in the monumental area of the historic city centre of Valencia, Spain. Notable buildings are grouped in the surrounding area, such as a cathedral, several churches, noble buildings and other notable buildings related to civil architecture, such as old endowments.

The building, known as the Palace of the Marchioness of Colomina (Palacio de la marquesa de Colomina), is located on Almudín Street No. 1. It is a large building that occupies the entire block, emphatic and prismatic, whose façades reveal its eclectic and, therefore, 19th-century origin. Its decoration is very austere and of classical type, being placed only on the windows and gates. The streets that delimit the building, in addition to Almudín Street to the south, are El Salvador Street to the east, and Weight of Flour Street, which adopts the shape of an “L” and to which the north and west façades fall. The closest monumental building to the palace is the great almudin of the city, located in front of the east façade of the Colomina Palace and separated from it by the Salvador Street. The almudin is a building of medieval origin, of modest construction, in these buildings “(...) the utilitarian aspect prevailed over the monumental value” [26], was intended for the sale of wheat that arrived in the city, and grew in size over the centuries to occupy its own city block.

The Weight of Flour Street alludes to a presently non-existing endowment. There are neither engravings, photographs of this building, nor publications that describe its appearance, even in the Historical Archive of Valencia, which guards medieval documents. However, it appears on several old maps of the city centre, the 18th-century plan by Tomás Vicente Tosca in 1704, and the Valentia Edetanorum vulgo del Cid by J. Fortea in 1738 [27]. In this sense, very few Valencians, seeing the name of the street, imagine that the Weight of Flour building existed, but no one knows what it looked like, and that, under the present eclectic decoration of the Colomina Palace (Figure 1), its original medieval façade remains.

The current appearance of the building, obviously, is not what it was in its medieval origins and is the result of its architectural evolution as a “house of the weight of flour”. In the 19th century, the building passed into private hands. Therefore, it had to adapt to its new use and the new fashion. The only existing documentation on the palace of the Marquesa de Colomina consists of two façade proposals, one of them not carried out (dated June 1863) (Figure 2a) and the one finally carried out (dated December 1863) (Figure 2b). Both were the work of the architect Manuel Ferrando, and their main difference is that the one in June has two floors and a semi-basement, while the one in December has a ground floor and two upper floors.

2.2. Data Acquisition

In order to support the following data merging operations between geometric and radiometric data, a complete 3D metric survey was necessary to obtain the geometric base which represents the starting point for the subsequent data fusion operations [28]. Hereby, another primary aim of the project was fulfilled: a proper documentation of the current building façades.

Therefore, in addition to the TIR acquisition, a complete terrestrial LiDAR survey was conducted. In fact, as reported in the previous section, although currently thermal images are increasingly used for 3D applications—for example, photogrammetric applications [29,30,31]—they are often not suitable for generating highly-detailed 3D surfaces due to their limited spatial resolution.

Additionally, a close-range photogrammetric acquisition was carried out, using a true high-resolution DLSR (digital single lens reflex) camera to obtain high-resolution colour orthophotos of the Palacio Colomina’s façades.

During the acquisition campaign, the following data was collected:

• 8 LiDAR scans, using a Trimble TX6 laser scanner (Figure 3a) in order to provide high-resolution point clouds to accurately describe the external geometries of the Palacio;
• 455 colour digital images, using a DSLR camera (model: Canon EOS 5D Mark II) (Figure 3b) to document the visible range of the electromagnetic spectrum;
• 308 TIR images, using a terrestrial thermal camera (model: FLIR ThermaCAM^TM B4) (Figure 3c) to detect anomalies or phenomena that are not visible to the human eye, but noticeable in long wave infrared (LWIR—8–14.000 nm).

2.2.1. LiDAR Scans Acquisition

The LiDAR survey was carried out to acquire the data of the external surfaces of the building in its current state (acquisition date: 9 May 2022). The main specifications of the laser scanner employed during the acquisition of the point clouds—a Trimble TX6—are reported in

Table 1. Main specifications of the sensors used during the acquisition campaign (Trimble TX6 laser scanner, Canon EOS 5D Mark II, FLIR ThermaCAM^TM B4).

Trimble TX6 Laser Scanner
Distance measurements	Phase shift
Wavelength	1.5 µm
Extended range	120 m
Horizontal and vertical range	360°/317°
Distance accuracy	<2 mm (1 sigma)
Acquisition speed	500.000 pts/sec
Camera	RGB
Canon EOS 5D Mark II
Sensor	CMOS 21.1 Mpx
Sensor size	Full frame (36 × 24 mm)
Image size	5616 × 3744 pixels
Focal length	24 mm
FLIR ThermaCAMTM B4
Thermal imager	Focal Plane Array (FPA), uncooled microbolometer
Focal length	45 mm
Image size	320 × 240 pixels
Spectral band	7.5 to 13 µm
Temperature range	−20 °C to +50 °C
Thermal accuracy	±2 °C, ±2%
Thermal sensitivity	0.08 °C at 30 °C

.

Considering the regular geometry of the Palacio, a small number of scans were sufficient to acquire all the external surfaces. One scan was acquired in the central part of each façade, and one corresponding to each corner to facilitate the subsequent registration procedures by providing an adequate overlapping between adjacent scans, requiring a total of 8 scans to cover all of the building.

2.2.2. Visible Colour Images Acquisition

Since the camera embedded in the used LiDAR system is characterised by a medium-low resolution, the RGB information provided by the laser scanner was not suitable for the generation of metric products characterised by an adequate radiometric quality. For this reason, a complete photogrammetric survey of the building was planned. The camera used during the photogrammetric survey was a Canon EOS 5D Mark II equipped with a 24 mm lens. The main specifications can be observed in Table 1. The acquisition of the visible digital images was performed following the well-known capture criteria (high convergence of cameras, overlapping between adjacent images, etc.). Due to the proximity of surrounding buildings on three of the four sides of the Palacio, the average acquisition distance was relatively short (4–5 m). This heavily affected the FoV (field of view) of the images and the portion of the façade covered by a single image. Only for the southern façade was it partially possible to increase the acquisition distance. For this reason, a high number (455) of images were necessary to cover the building’s surface. The images were collected with the following camera parameters: diaphragm aperture f/11, ISO-100, and exposure time depending on the light conditions of each façade (approximately between 1/80 s and 1/200 s).

2.2.3. TIR Images Acquisition

A critical issue related to the acquisition of TIR images in an outdoor environment—where it is barely possible to influence the environmental conditions, such as temperature and weather—is represented by the necessity to acquire the data with an optimal thermal response. Generally, these kinds of images need to be acquired during sunrise or sunset, when the thermal equilibrium is interrupted, and a significant change in the thermal conditions is observed due to exposure to sunlight [16]. However, in the case of the current research, the Palacio was surrounded by many buildings of an equal height or higher. For this reason, the acquisition of the TIR images was planned at different times for each façade, considering the sun position (Figure 4), to capture the thermograms with the optimal sun exposure and shadow conditions and, consequently, to optimise the thermal response. The thermal camera employed was a FLIR ThermaCAM^TM B4 (the principal specification of this terrestrial thermal camera can be observed in Table 1).

As well as the acquisition of the visible images, the presence of neighbouring buildings represented one of the crucial problems encountered during the TIR image acquisition operation. As mentioned above, the criticality is related to the reduced acquisition distance (4–5 m) (Figure 5a) due to the proximity of other buildings (concerning three out of four sides of the surveyed monument). Additionally, this difficulty was further accentuated by the camera’s limited FoV, considering the relatively long focal length (45 mm, significantly higher compared with the 24 mm focal length of the lens equipped on the DSLR camera). As a result of the described acquisition conditions, each image covered only a restricted and very limited portion of the considered façade (Figure 5c). Additionally, as well as the previous photogrammetric acquisition, the images of the higher part of the building were considerably foreshortened and characterised by a high perspective distortion (Figure 5b).

For this reason—and due to the difficulties in acquiring the TIR images following the well-known photogrammetric criteria as high convergence, high overlapping, etc.—only a portion of the dataset was collected with the aim of using it for photogrammetric purposes. In particular, the photogrammetric acquisition was conducted for the southern façade, the only façade where it was possible to increase the acquisition distance due to the partial absence of surrounding buildings, with the exception of a small building on the left side. This was supplemented by some very localised phenomena, e.g., a portion of the northern façade, where a very high thermal discrepancy was observed between the second and the third floor.

2.3. Data Processing

The acquired data was processed as follows:

• The LiDAR scans were registered using an automatic plane-based registration;
• The colour digital images were processed following a photogrammetric SfM-based procedure;
• The TIR images were processed following two different pipelines: when possible (i.e., when the thermograms were acquired following photogrammetric criteria), a standard photogrammetric SfM-based workflow was followed, similar to colour processing; the second workflow consisted of projecting the thermal images on the 3D mesh derived from LiDAR—characterised by a very high spatial resolution—to generate a thermal texture;
• These procedures will be further described in the following sections. The workflows carried out in this study can be observed in Figure 6.

2.3.1. Terrestrial Laser Scanning Processing

The 8 scans were registered in the same local reference system using a procedure based on automatic plane-based registration. The software used for the alignment of the scans was Trimble RealWorks v. 11.3 (Sunnyvale, CA, USA). The mean discrepancies among the registered scans at the end of this operation were millimeter-level (an overall value of approximately 2 mm was achieved, consistent with the high overlapping between the adjacent scans). During the registration procedure, it was also possible to colourise the point clouds using the low-resolution visible colour images acquired by the integrated HDR camera (10 Mpx resolution) during the acquisition of the scans.

Following this operation, the 8 point clouds were merged and imported into the Cyclone 3DR platform v. 21.3.6.39469 (Wetzlar, Germany). Then, the merged point cloud was filtered to remove noise, outliers, and redundant points. The final point cloud was composed of about 60,000,000 points.

2.3.2. Visible Colour Image Processing

Before proceeding with the processing of the photogrammetric data, a set of points was extracted from the registered LiDAR point cloud. These points were used as GCPs (ground control points) and ChPs (check points) with the aim of: (a) registering the photogrammetric point cloud in the same local reference system of the LiDAR data, and (b) evaluating the metric accuracy of the final results after the bundle block adjustment. The colour images were processed using the software Agisoft Metashape v. 1.8.0. A standard photogrammetric SfM-based approach was followed [8,32]:

• Internal camera orientation;
• Relative external orientation and tie points generation;
• Absolute external orientation using GCPs;
• Evaluation of the metric accuracy using ChPs.

The 455 images were successfully oriented. The RMSE (root mean square error) observed after the bundle block adjustment was 0.007 m—concerning the GCPs—and 0.011 m—concerning the ChPs—which was coherent with a representation scale of 1:50/1:100.

Considering the type of building, corresponding with plastered walls, the extraction of homologous points was lower due to the absence of noticeable features and characterisation of the masonry. It was possible to observe a high noise level in the final dense point cloud (obtained after the generation of the depth maps) and even gaps and holes in the point cloud.

2.3.3. TIR Images Processing (Photogrammetric SfM-Based Workflow)

Regarding the photogrammetric SfM-based approach [8], the thermal images were processed using the same workflow described in the previous section. Additionally, in this case, the images were oriented using Agisoft Metashape. As well as for the visible colour images, the set of points used as GCPs and ChPs for evaluating the metric accuracy were extracted from the LiDAR point cloud. Concerning this last aspect, a well-known issue related to the use of reference points (both GCPs and ChPs) for photogrammetric applications involving thermal images is connected to the difficulties in the unambiguous identification of the traditional markers—often made of paper or plastic—that are usually affixed on the surveyed surfaces and measured using conventional topographic methods. This critical issue is often solved using markers composed of low-emissivity materials (for example, aluminium, characterised by an emissivity of ≈0.04–0.09) [32], retroreflective markers (formed of reflective foil) or thermally induced active markers with an electrical resistor as artificial heat source in its centre [16]. However, as reported in Patrucco et al. [21], it is also possible to use points extracted in those areas where the radiometric contrast between adjacent materials and elements with different emissivity allows the unambiguous recognition of the selected points even in the absence of artificial markers. This strategy was applied during this research experience by exploiting the recognisable natural points, e.g., the corners of the windows.

Before proceeding with the photogrammetric processing, the raw thermograms were pre-processed and converted to panchromatic single-band images (TIFF format, 32-bit floating point). After this procedure, the digital number embedded in each pixel indicated the temperature value.

As mentioned in the previous sections, it was only possible to entirely reconstruct the southern façade, while—regarding the other sides of the building—only small and localised portions were digitally recorded using this method. In Figure 7a,b, it is possible to observe the sparse cloud of tie points generated after the orientation of the cameras. In the case of the southern façade dataset (21 images), the average acquisition distance was approximately 20 m, while the estimated average GSD (ground sample distance) was ca. 5.3 cm. Instead, the photogrammetric datasets acquired on the other sides of the Palacio (e.g., the dataset of the northern façade upper area, composed of 18 images) were collected from an average acquisition distance of about 5–7 m, while the estimated GSD was ca. 1.5–2 cm. Regarding the southern façade, the RMSE (root mean square error) after the bundle block adjustment was 0.009 m—concerning the GCPs—and 0.033 m—concerning the ChPs; regarding the upper area of the northern façade, the RMSE observed on the GCPs was 0.004, while the RMSE observed on the ChPs was 0.015 m. These values were higher than those observed during the processing of the visible dataset, but they were coherent with the lower resolution of the TIR images—affecting the manual collimation of the points—and the higher GSD.

2.3.4. TIR Images Processing (Texture Projection)

As anticipated in the previous section, it was not possible to perform an effective and efficient data fusion between the TIR images and the other acquired data for all surfaces of the external walls of the Palacio. For this reason, it was necessary to develop a second strategy.

In this case, the relative orientation between each thermal image and the high-resolution 3D mesh derived from the LiDAR point cloud was manually estimated by collimating homologous points recognisable on both the TIR image and the corresponding area of the three-dimensional surface.

The different steps of the followed strategy were as follows:

• Preliminary selection of the TIR images. In this case, ca. 60 TIR images were selected—from those collected during the different acquisitions—with the aim of covering all the façades of the building;
• Pre-processing of the thermograms. During this preliminary phase, the thermal images were pre-processed. Starting from the raw panchromatic image, where the digital number embedded in each pixel represented the detected absolute temperature, each thermogram was converted into a rendered .jpg where the temperature range was represented through a predefined false colour palette. For each façade, the selected set of images was exported after setting the same colour palette and temperature range (southern façade: 32.0–23.0 °C; eastern façade: 34.0–29.2 °C; northern façade 27.0–14.0 °C; western façade: 26.0–18.0 °C);
• Manual collimation of the homologous points. A set of points was manually identified on each thermogram and the corresponding area of the 3D mesh, to estimate the orientation matrix of each image [33] (Figure 7c);
• Generation of a thermal texture. From the previous mosaicking procedure, where all the TIR images were oriented, a thermal texture was generated and projected onto the 3D mesh (Figure 7d);
• Editing of the texture. The generated texture was edited to fix some misprojected areas, fill any gaps and lack of information, or refine the edges between the adjacent images.

For the pre-processing of the thermal images, the open-source software Wrapper was used, while the operations related to the generation of the thermal texture were carried out using the Cyclone 3DR platform.

3. Results

3.1. 3D Data Fusion for Enriched Models Generation

Since both the colour images and TIR images were oriented in the same local reference system of the LiDAR point cloud, it was possible to integrate the radiometric information with the geometries by projection. In this case, the LiDAR point cloud allowed the achievement of better results in terms of geometric resolution and level of detail (Figure 8, Figure 9 and Figure 10). Therefore, it was used as the reference geometric surface for the reprojection of the radiometric data. It should be noted how, in this case, the photogrammetric approach using the colour dataset was necessary to provide high-resolution radiometric information to the LiDAR point cloud. As already stated in the previous sections, the camera sensor embedded in the most of laser scanner devices was characterised by a medium-low resolution.

In summary, due to this strategy, it was possible to achieve:

• TIR-colourised LiDAR point cloud;
• True colour-colourised LiDAR point cloud;
• High-resolution TIR-texturised 3D mesh;
• High-resolution true colour-texturised 3D mesh;
• High-resolution TIR orthoimagery;
• High-resolution true-colour orthoimagery.

Regarding the second workflow used to generate TIR orthophotos, it should be noted that exploiting the mosaicking procedure for the generation of a thermal texture was a generally less precise solution, in comparison with the photogrammetric approach. Furthermore, it represented a time-consuming alternative, since it required significant manual involvement of the operator. In contrast, in the case of the SfM-based approach, the homologous points were automatically matched during the keypoints extraction phase, leading to an automatic orientation of the images.

Additionally, the metric products—specifically the orthoimagery—derived from this strategy, were represented by rendered images where the temperature was expressed by a pseudocolour palette and not by the digital number; contrary to the orthomosaic derived from the SfM-based approach.

However, it should be noted how the second workflow was more flexible, and how it was possible to be applied to the building in its entirety, allowing the generation of high-resolution orthoimagery of all the four façades of the Palacio de Colomina (Figure 11), while the photogrammetric pipeline allowed the achievement of good results for only some portions of the surveyed façades.

3.2. Parametric Modelling

In a parametric modelling applications framework, a building is divided into volumetric elements with identical parameters [33,34]. The composition of these individual parametric objects represents a building in a building information model (BIM) or historic building information model (HBIM). Within this HBIM environment, combining different metric data—e.g., point clouds derived from laser scanning technologies or photogrammetry—and archival information of a historic building in a layer-based system, is possible. In addition, it is possible to automatically create sections, elevations, and details of any part of the building. A further powerful application of this technique is represented by the possibility of producing a timeline with different epochs represented by individual 3D models related to transformations of the building during its life cycle. As a result, it is possible to visualise and analyse the volumetric changes of a building. Furthermore, orthographic projections and 3D models (such as wireframe and textured models) can be linked and visualised in the HBIM [35]. Additionally, to provide a comprehensive representation of the architectural heritage, it is also mandatory to include as much information related to its history, condition and environment as possible [36]. Such information and further sources can be linked to the single elements inside the HBIM. In this way, HBIM is a powerful tool for the combination of metric and geometric data with archival sources, thus, making it useful for the presentation, management and monitoring of historic buildings [37,38].

In this study, HBIM combined the individual data from terrestrial laser scanning, photogrammetry, thermography and existing architectural plans in a digital database. The BIM platform Autodesk Revit 2021 was used. First, the parametric 3D model of the outer façades of the Palacio was built based on the geometric data of the LiDAR point cloud.

After the modelling phases in Revit, the documents related to each façade were added and archived in the parametric model. In addition to the LiDAR point cloud, orthoimages of the thermography of each façade, the historical plans from 1863 and the condition mapping of 1995 were metrically corrected and imported as individual layers (

Table 2. Overview of the Palacio de Colomina façades considering the parametric model, the primary data used during the modelling phases (e.g., LiDAR point clouds and thermal orthoimagery), and the historical sources.

	Parametric Model	LiDAR Point Cloud	Thermal Orthophoto	Historical Plan, 1863	Condition Mapping, 1995
North façade
Eastfaçade
South façade
West façade

).

4. Discussion

With the aim of performing a comparison between the achieved point clouds (LiDAR point cloud, photogrammetric point cloud derived from visible colour images, photogrammetric point cloud derived from thermal images), a density analysis (by means of the number of neighbours approach) was performed using the open-source software CloudCompare v. 2.11 Alpha. In this analysis, a sphere (with a radius defined by the user) was computed for each point, and the number of points inside the sphere were measured. In this case, the selected radius was 0.2 m.

As it is possible to observe in Figure 12 and Figure 13, the denser and more detailed point cloud was the LiDAR point cloud, even after the decimation process (as expected, the point cloud was denser in those areas closer to the laser scanner stations during the acquisition, specifically the centre of the façade and the lateral areas, near the corners). The level of detail was appreciable also in the visible colour photogrammetric point cloud, although it was possible to find many areas characterised by high levels of noise due to the presence of surfaces (explicitly corresponding to the plastered walls) that represented a challenge for the photogrammetric approach (vid. Section 2.3.2).

Concerning the photogrammetric data derived from the SfM-based processing of the TIR images, from a preliminary visual inspection, it appears evident that the thermal point cloud was characterised by a lower level of detail and a high level of noise, coherently with the limited spatial resolution of the TIR images. This aspect becomes even more evident by comparison with the LiDAR point cloud, which can be observed in Figure 12. Moreover, as it is possible to follow in Figure 13c, the thermal point cloud was significantly less dense than the others.

It should also be noted that in both the colour point cloud and the thermal point cloud, the left side was more poorly reconstructed (presenting holes, gaps, and lack of information). This was due to the fact that, as previously mentioned, the left side of the Palacio was partially covered by a small building that prevented, in that area, increasing the image acquisition distance. For this reason, the images on the left side of the 3D model were less convergent, leading to a noisier and less complete 3D reconstruction.

On this basis, it can be deduced that, individually, the techniques that were used for the 3D documentation of the Palacio de Colomina were not sufficient to achieve all the research aims. This was because of difficulties in generating a high resolution and topologically correct 3D model from both the visible colour dataset and the thermal dataset, due to (a) the challenging acquisition conditions (regarding both datasets); and (b) the intrinsic low spatial resolution of the TIR images [39]. The topological errors observed in both point clouds (Figure 13b,c) can lead to miscalculations in the reprojection and, thus, affect the generation of orthoimages.

In summary, the most efficient strategy is represented by the possibility of integrating the data from the different sensing techniques to obtain a semantic, multi-layered, value-added, metric product. As noted in the previous sections, the integration of the TIR images and 3D surfaces allowed us to spatially connect non-visible phenomena which are not easily detectable using traditional sensing methodologies, or images acquired in the visible range of the electromagnetic spectrum. In the current research, the surfaces generated from the LiDAR point cloud—characterised by a high spatial resolution—helped minimise the effects of an inaccurate projection of the thermograms, since the geometries derivable from TIR imagery are characterised by a high number of topological errors when reconstructing 3D models.

It is necessary to note how the described strategies allow metrically measurable phenomena to be obtained, that assist in comparing and interpreting built assets belonging to the cultural heritage. The three-dimensional spatialisation of the information connected to the TIR radiometry enhanced the ability to interpret and understand traces, discrepancies and anomalies, enabling investigations to also consider the connections of architectural elements and volumes—also from a semantic point of view—that no longer exist today. This aspect becomes evident in the case of the Palacio de Colomina, where the integrated analysis of the produced thermal metric products (3D models and orthoimagery) allowed us to notice several thermal discrepancies coherent with a different conformation of the original building.

In particular, these pieces of evidence highlighted two types of aspects that deserve to be emphasised and deepened:

• Traces that are consistent with the presence of architectural elements obliterated by subsequent building interventions (Figure 14);
• Possible clues of drastic modifications regarding the volume of the building.

In the first case, by way of example, we report some discrepancies detected in the central part of the eastern façade. Hereby, radiometric contrasts coherent with the presence of (a) façade decorations, niches, or obliterated windows, and (b) remains of the decorated frames of the central windows, appear more massive and thicker, compared with the current ones (Figure 14, areas A and C).

In this case, both the adopted strategies aimed at integration of LiDAR data and TIR images (SfM photogrammetry and texture projection, Figure 15) enabled us to spatialise the hidden elements on the 3D surface, representing a significant advantage during the 3D modelling phases.

However, for a correct interpretation of the anomalies to distinguish between niches and obliterated windows, a comparison with archive data helps. To initialise modifications and renovations of the Palace, CAD plans with condition mapping were created in 1995. On the eastern façade, two glass housings of the central balconies are shown (Figure 16). Such housings were typical in Valencia during the first half of the 20th-century and are still preserved in many other buildings. They were used to enlarge the room by including the balcony. Therefore, the wall was opened to the entire width of the balcony. The shape of anomalies A and C detected in the TIR data (Figure 14) well fits the frame of the wooden housing shown in the plans. Two more pieces of information were given in the plan. First was the reason why the housings were removed during the renovation, as their construction was indicated as being in poor condition “agrietamiento carpinteria”. Second was the former opening of the wall to the entire width of the balcony, mentioned by a horizontal cutout of the wall “Degüello horizontal muro”. This means the housings were removed during the renovation, and the openings in the wall were closed to fit the window frames. Therefore, other materials were used, causing the thermal differences.

In order to stress the effectiveness of the possibility of matching historical documents in a BIM environment, an example is proposed regarding the thermal anomalies detected on the east façade. While overlaying the thermal information and the historical plans, a direct comparison was much easier, and the location of the thermal discrepancies “A” and “C” could be analysed in connection with the condition mapping of 1995 (Figure 17).

The cause of the thermal anomaly “B” remains unclear. The discrepancy shape is coherent with a horizontal line ending in an arch. Within the HBIM, it is possible to model virtual reconstructions. In this way, the thermal anomaly was reconstructed together with the changes in the central windows (Figure 17c). Based on this virtual reconstruction, the anomaly can be better discussed. There are two possible interpretations: in the first, the anomaly could be interpreted as a niche in the façade, which is common in similar examples in Valencia (Figure 18). However,, as observed in the virtual reconstruction, the niche would not fit symmetrically into the façade, shifting out of the central axis. Therefore, a connection to the visible façade representing the construction of 1863 with perfect symmetries is unlikely. This leads to the second interpretation. The anomaly may have been caused by an occluded window of an earlier construction of the building. This would be an indication that structures of previous buildings are still preserved underneath the shape of the 1863 façades.

Additionally, another thermal discrepancy was detected, suggesting a modification of the building volume, as observable in Figure 19. In this case, a significant difference in temperature (ca. 5–6 °C) was observed between the second and the third floors of the northern and the western façades (Figure 13 and Figure 19). The phenomenon appears consistent with an enlargement of the Palacio with an additional floor using different materials, characterised by a significant emissivity difference. Together with the detection of the hidden arch (Figure 17), this observation proves the existence of two levels of a preliminary building, preserved under the actual façades. This is coherent with the information derivable from the 18th-century plan from Tomás Vicente Tosca in 1704 and the Valentia Edetanorum vulgo del Cid from J. Fortea in 1738 [27] where the building was shown with only two floors and a smaller volume. Furthermore, the integration of former constructions could be the reason why the first proposal for the renovation in 1863 (Figure 2a) was rejected. The first draft from June 1863 was characterised by only two floors, but it included a basement. If the preliminary building did not have a basement, completely new construction would have caused much higher effort and costs, especially compared with the re-use of existing constructions that were enlarged with a third floor on top.

5. Conclusions

This paper compared two strategies for integrating TIR images with LiDAR and visible colour data. From the comparison, it was possible to observe that no solution outperformed the other. This aspect underlines that careful planning of documentation activities is always mandatory in the field of conservation and restoration of cultural heritage. Every asset belonging to built heritage is characterised by specificities and peculiarities. Therefore, customised solutions are often required to fulfil the documentation needs related to cultural heritage.

The integration of TIR images and high-resolution 3D data appears to be extremely advantageous for modelling purposes, allowing the generation of orthoimagery, colourised point cloud and texturised 3D mesh. These value-added metric products represent a solid basis for parametric modelling procedures, enabling spatialisation of the thermal anomaly and, consequently, favouring the interpretation of the former volume of the Palacio.

Additionally, the opportunity of HBIM strategies should be noted—not only for the documentation of built heritage, but also for its management and analysis. In fact, this increasingly popular technology has demonstrated that the informative models derived from reality-based data—such as point clouds, 3D mesh, orthophotos and archival sources—not only need to be used as standalone data, they can be merged in an HBIM to provide a powerful tool for heritage experts. Such an information system allows the management of multi-disciplinary analyses of the considered buildings [40,41,42].

Furthermore, as presented in the case study of the Palacio de Colomina, this method enables the management of multi-scale information with different levels of detail [43,44] depending on the granularity levels of the considered data. This has allowed the analysis of the Palacio from two different points of view: a volumetric point of view, where remote sensing data were overlayed with historical plans trying to digitally visualise, compare and reconstruct the different constructive phases of the building; and a more interpretative analysis of the individual architectural elements, that necessarily require a higher level of detail to be studied.