Multitemporal Thermal Imagery Acquisition and Data Processing on Historical Masonry: Experimental Application on a Case Study

Abstract

The recent improvement of infrared image quality has increased the use of thermography as a non-destructive diagnostic technique. Amongst other applications, thermography can be used to monitor historic buildings. The present work was carried out within the framework of the Horizon 2020 European project SHELTER, which aims to create a management plan for cultural heritage subject to environmental and anthropogenic risk. Among the chosen case studies is the Santa Croce Complex in Ravenna (Italy), which is exposed to different hazards, including flooding. The church has a peculiar architecture that develops below the street level, so the internal walls are affected by the deterioration caused by rising humidity. In such a case of advanced degradation, passive thermography cannot be used to its full potential. For this reason, an innovative methodology involving active thermography was first developed and validated with laboratory tests. Secondly, we conducted its first application to a real case study. With this purpose, an active thermography survey with forced ventilation was carried out to enhance different stages of material degradation by means of automatic classification of multitemporal data. These experiments have resulted in a method using an active thermal survey in a high moisture content environment to detect masonry degradation.

1. Introduction

Nowadays, the monitoring and conservation of cultural heritage sites are topics of great interest addressed from different points of view. In this context, it is important to find new integrated approaches to detect possible risks for historical buildings, to map degradation phenomena and act to better conserve or restore assets [1]. Historical buildings are complex systems subject to risks deriving from internal and external causes. It is necessary to carefully monitor the internal microclimate of the buildings and atmospheric pollution to preserve original materials [2,3,4].

The European SHELTER project (Sustainable Historic Environments hoListic reconstruction through Technological Enhancement and community-based Resilience), financed by the Horizon 2020 Research and Innovation Programme, aims to develop new resilience practices for cultural heritage subject to risk factors due primarily to climate change [5]. The five open labs selected as part of the SHELTER project represent different heritage typologies, from small to large scale. According to the objectives of the project, the Santa Croce Complex in Ravenna was chosen as an urban open lab. The Santa Croce Complex, located in the area of San Vitale (UNESCO site since 1996), is composed of the church (5th century) and the archaeological area in the surroundings (Figure 1). As part of the project, a multidisciplinary study was carried out to analyse all the risk factors of the complex [6]. A critical point was to establish a reliable geometric knowledge base, derived from the application of a rigorous survey integrating many geomatic techniques.

To complete the survey activities held by the Geomatics Group of the Interdepartmental Centre for Industrial Research in Building and Construction and of the Department of Civil, Chemical, Environmental and Materials Engineering (DICAM) of University of Bologna [7], an active thermal survey was carried out. A previous survey campaign involved the integration of different techniques (laser scanner, terrestrial and aerial photogrammetry, topographical surveys by total station and GNSS) to build a three-dimensional model of the Church of Santa Croce. This 3D model was the base for developing an information model with a Scan-to-BIM approach. Moreover, the 3D model tied to the global reference system (UTM WGS84) was used to georeference all other data acquired for the study. The thermal survey described in this paper will allow completion of previous activities, adding new information on the state of the masonry.

The architectural conformation of the church is, to some extent, related to a huge problem of rising humidity. The floor of the church, now almost completely lost, is currently about 2.70 m below the street level (about 1.40 m above mean sea level). From the geomatic surveys carried out, the floor of the Church of Santa Croce stands at 1.30 metres above sea level. Moreover, considering that the groundwater level is between 0 and −5 metres, the area is affected by frequent flooding, which increases the degradation of the masonry. In addition, the Ravenna area is notoriously affected by a phenomenon of subsidence, due to natural and anthropogenic causes [8]. In the context of cultural heritage, comprehensive investigations aimed at assessing and monitoring damage due to these risks is crucial for the preservation and maintenance of the existing asset.

Due to the particular structure of the Church of Santa Croce and its historical and cultural importance, it is necessary to use appropriate survey methods. In order to keep the historical building intact, non-destructive and contact-free diagnostic techniques are mainly used [9,10,11]. Among the different non-destructive techniques, thermography has been increasingly used for the analysis of materials and building defects in the last decades [12,13]. In the interior of Santa Croce, due to high presence of humidity and significant presence of moss on the surface of the masonry, it is difficult to obtain appreciable information with passive thermography only. Active thermography is extremely useful in cases where the temperature contrast is hard to recognize [14]. Active thermography allows much more information to be acquired as it studies the behaviour of the detected object during a defined time period. Moreover, image processing and data mining techniques can be successfully applied to this type of dataset to produce informative maps. Indeed, image classification helps the transformation of the collected multitemporal images into meaningful and easy to interpret information about the investigated surfaces.

This paper describes the methodology used for active thermal surveys in environments with high humidity. This approach was first applied in the laboratory, where the humidity conditions of Santa Croce walls were reproduced, and then in situ. A fan was used to ignite evaporative flow, which is weak in the environmental conditions inside Santa Croce due to lack of apertures that favour air currents. Thanks to the ventilation, it was possible to study the behaviour of three areas: very wet, wet and damp. Thermal images were acquired even after the fan was shut down to analyse the trend of water evaporation from the wall. The aim of this work was to detect anomalies in coating materials, which cannot be detected by passive thermography. The main purpose of this survey was to map the degradation present on the inner covering of Santa Croce Church.

2. Materials and Methods

Infrared thermography (IRT) is a non-destructive technique often used to map moisture distribution in historic and modern buildings [15]. Detecting the presence of moisture is particularly important, as it is one of the principal causes of building pathologies [16]. The two main approaches that can be used to perform a thermal survey are passive and active. Passive (or static) thermography can detect the thermal radiation emitted by the body without changing the natural conditions. The use of active (or dynamic) thermography, on the contrary, involves the use of an instrument, which depends on the thermophysical properties of the test object, that may highlight some behaviours detectable with the thermal camera [17]. Often, in active thermal surveys, external heating sources (halogen lamps, flashes, infrared light) are used to activate heat dissipation [18] or air jets to activate the evaporative flow in materials where moisture is present. Moreover, depending on the material, other methods such as electromagnetic induction or ultrasonics can activate mechanisms that can be detected with the thermal camera [19].

The walls of the Santa Croce Church are affected by visible damage caused by humidity. The interior of the church has a relative humidity (%RH) in the order of 80%, and the lower parts of the walls are covered by a layer of moss that rises from the floor level. In this case, passive thermography cannot provide information better than that observed by the naked eye. Through a passive thermal survey, it was only possible to identify qualitatively the coldest bands, while it is not possible to see small variations within them (Figure 2).

To create a more accurate degradation map, an active multitemporal thermal survey was planned with multiple acquisitions at specific time intervals and the use of a fan to force the ventilation of the investigated area. A preliminary test was carried out in another site on a plastered wall with signs of degradation due to humidity in order to develop the methodology for data acquisition and processing. A thermal camera FLIR P620 was used, both for the preliminary test and in situ. The FLIR P620 has a spectral range between 7.5 μm and 13 μm and a 640 × 800 image resolution; the camera was equipped with the standard 24° lens.

For image processing and data analysis, commercial software ENVI 5.6 [20] and open-source QGIS 3.16 [21] were used for different purposes. Specific visualization methods were employed to investigate changes more effectively, using RGB composite imagery to jointly analyze the different states. Spatial co-registration operations were needed to align multiple images acquired over time in a single multi-band stack, and Principal Component Analysis was finally performed on these datasets.

In order to frame the survey within an absolute reference system, a geometric survey of the area of interest was also carried out with a volumetric scanner. The instrument used was the F6 smart scanner, distributed by Stonex, which projects structured light patterns in the near infrared (wavelength 850 nm) to produce an accurate 3D model of the area through reconstruction from the image series acquired. In the situation described, the instrument precision is at the mm level. An RGB sensor provides the ability to associate natural colours with the three-dimensional point-clouds generated. From the survey, a point-cloud of about 20 million points was produced. The survey performed with the F6 scanner was aligned with the survey previously conducted by the Geomatics group with a laser scanner, photogrammetry, Total Station and GNSS [7].

3. Results

3.1. Preliminary Test

The wall for the first test was divided into three zones with adhesive tape to assess the behaviour of the material depending on the distance of a fan placed to the left. The test was performed in an area of 1.20 × 1.20 m (Figure 3). Porcelain stoneware targets of 3 × 3 cm with retro-reflective adhesive tape on the front were used. During the test, temperature and relative humidity values were recorded by a data-logger.

Water was sprayed on the wall to reproduce, at least partly, the three different moisture levels identified through passive thermography carried out previously on the Santa Croce masonry. The test was conducted with three different situationss: dry wall, damp wall and wet wall. For this purpose, 50 mL of water was sprayed to reproduce the damp wall. Then, another 50 mL was added to produce wet wall. In this way it was possible to evaluate how the active thermography results depend on the level of humidity of the surface. The test was performed capturing one image every 20 s for three minutes with the fan on, then capturing every 20 s for another five minutes with the fan off. After acquiring the main sets (dry, damp and wet wall) the test was repeated after 1 h. Then, thermal images were acquired every 20 min for 4 h to record information concerning evaporation under natural conditions, without the use of a fan.

To better visualize and analyse the thermal image sets, and to show differences over time, multitemporal composite images were produced as shown in Figure 4. For each set, three images were chosen that were representative of the active thermal survey, and then assigned to RGB channels. The start of the test is represented with the red channel, the last acquisition (with fan on) with the green channel, and the end of the test (fan off) with the blue channel. This configuration for the RGB composite was used for each acquisition set: (Figure 4b) dry wall, (Figure 4c) dump wall, (Figure 4d) wet wall., (Figure 4e) after 1 h from wetting.

The results of the test demonstrate how the use of active thermography can highlight degraded areas otherwise not detectable with passive thermography. Regarding the last set of images acquired (one every 20 min for a total of 12 images) with passive thermography, the RGB composite is represented with: t₀ red channel, t₆ green channel, t₁₂ blue channel (Figure 4f). The acquisition of thermal images during the evaporation phase allows identification of those areas where the water evaporates faster, a factor that could detect other defects, such as raised plaster.

3.2. In Situ Test

The thermal survey inside the Santa Croce Church was carried out with the FLIR P620 thermal camera, fixed on a tripod, following the same scheme developed during the preliminary test. Images were captured in two phases, the first phase involving the use of the fan, while in the second phase the fan was switched off. However, due to the characteristics of the masonry and the presence of moss on the surface, it was necessary to use a different system for the positioning of the targets needed to share a common reference system with other surveying data. Poles slightly separated from the wall were used to fix retro-reflective porcelain stoneware targets. Before the start of the test three different bands were identified: the lower band (zone 1), almost completely covered by moss, an intermediate transition band (zone 2), and an upper band without moss (zone 3). As in the preliminary test, images were acquired for each zone corresponding to different degrees of humidity. The wall was ventilated by a fan positioned at three different heights. During the entire survey, temperature and relative humidity were continuously acquired through a data logger. The acquisition scheme was acquisition of 10 images (1 every 20 s) with fan on, and acquisition of 20 images (1 every 20 s) with fan off. A total of 90 thermal images of the northern wall were captured during the survey.

As already mentioned, a precise 3D geometric survey of the area of interest was performed by a handheld structured light projection scanner, and a point-cloud of about 20 million points was inserted in an absolute spatial reference system thanks to a previous multi-technique geomatic integrated survey. In this way it was possible to georeference the thermal datasets in the global reference system thanks to the references of the poles surveyed with the volumetric scanner.

Given the irregularity of the wall, the 3D survey of the reference area carried out with the F6 scanner was also used to evaluate possible correlations between humidity and geometric characteristics. To present the results more clearly, a best-fit plan representative of the wall surface was calculated [22]. The best fit plane was calculated excluding the poles, used as reference and to position targets. To obtain a scale of values in meters representing the positive or negative deviation from the best-fit plane, the point cloud was translated in the z-axis of the best-fit plane value. In this way, the zero-value indicates that there is no deviation from the average z given by the best-fit plan. Negative values (blue) indicate indentations, while positive values (green) indicate protrusions. (Figure 5).

The obtained data show that the deviation of the bricks from the best fit plane was significant. It is therefore necessary to integrate the results obtained from the thermal data with this geometric data. The geometric characteristics can influence evaporative flow, accentuating or attenuating the humidity problems present in the whole structure.

3.3. Data Preprocessing

The images captured through the thermal camera were edited with FLIR proprietary software. After downloading the temperature and relative humidity data, they were cross-referenced with image capture time to assign the correct values to each thermal image. The IR images were then exported in tiff 32-bit format.

An IR image can be seen as a matrix of pixels providing information about temperature T(x,y), where x is the number of pixel column and y is the number of the row.

The IR images were then spatially co-registered with each other (GIS software was used for this purpose), in order to have a perfect match of the corresponding pixels in all the images referred to each of the three zones (Zone 1: bottom, Zone 2: middle, Zone 3: top).

For each Zone, a multi-band file was created from the available IR images by a layer stacking operation, with a shared spatial grid. The data were then stored in a 3D matrix in which each pixel could be identified by T(x, y, t), where x is the column coordinate, y the line, and t the temporal coordinate. For each pixel identified with the position (x, y), it is possible to reconstruct the temperature time series. T(x, y, 1) refers to the first acquisition, T(x, y, 10) is the image collected at the end of the ventilation phase and T(x, y, 30) is the last acquisition (Figure 6).

To allow a comparative temperature analysis and to highlight the temperature, for each zone, a new multi-band stack was subsequently produced. All TIR images, pixel by pixel, were divided by the first image acquired at t(1), at the test beginning, as follows:

$$T_{ratio}\left(x,y,z\right)=\frac{T\left(x,y,t\right)}{T\left(x,y,1\right)}$$

(1)

The temperature ratio (T_ratio) allowed comparisons of increase or decrease of temperature with respect to the first acquisition conditions. The time series of T_ratio was reconstructed for pixels thanks to the ENVI LayerStack tool.

3.4. Data Processing

A clustering method was used to extract valuable information from these data. Clustering is a process allowing the identification of natural groups (clusters) in a multidimensional dataset [23]. Among data mining algorithms, grouping techniques are supposed to find the most homogeneous clusters that are as distinct as possible from other clusters: maximizing inter-cluster variance while minimizing intra-cluster variance [24]. In other words, these algorithms should automatically recognize patterns intrinsically present within the dataset [25].

In image processing, data clustering is often known as unsupervised classification. Indeed, grouping pixels with the same behaviour in time was performed without selecting a priori any labelled training for the algorithm. In this case, the objective was to identify areas of the surface that responded to venting in the same way. One of the most widely used clustering methods is the K-means algorithm. Its simplicity, efficiency, and empirical success has made it suitable and popular for a large variety of applications and disciplines [26]. K-means is an iterative unsupervised classifier that minimizes the intra-class distance [27]. Once fixed, the parameter k, i.e., the number of classes in which the dataset must be partitioned, k centroids are at first selected randomly in the feature space and initialized. Centroids, also known as seeds, are representative of the centre of the cluster Pixels are assigned to the nearest centroid and then the mean value for each class is computed to detect and update new centroids. The iterative process ends when it arrives at convergence.

One of the well-known issues with K-Means [28,29] is its initializing dependence. According to Xu et al. [30], K-Means initialization can be improved by searching the centroids in the Principal Components feature space. Principal Component Analysis (PCA) consists of the transformation and the reduction of the initial feature space, representing data as a linear combination of a new uncorrelated system of variables, the principal components, maximizing variance in the dataset [31].

For each T_ratio stack, PCA was performed (Figure 7). Only the first 10 T_ratio images were considered to obtain the new feature space. The covariance matrix was calculated and the first third component results having eigenvalues greater that zero. For each Zone acquisition set, the first three component were selected as significant. A K-means unsupervised classification was applied to the stack of the PCs. The algorithm automatically identified five classes, with 1000 max iterations and a change threshold of 5%.

The investigations carried out showed that active thermography is able to reveal more information about the masonry than passive thermography. As can be seen from Figure 8, in the images taken at the end of the test, the detected area appears homogeneous, as the effects of ventilation tend to dissipate. To highlight the contrast with active thermography, a composition of multi-temporal images was created in which the key moments of the active test can be visualised.

The K-Means unsupervised classification allowed identification of five classes of T changes due to fan action. The temperature ratio allowed easy comparisons of increases or decreases of temperature with respect to the first acquisition condition. The average T ratio for each class was calculated. Moreover, some Points of Intertest (POI) were selected for each class in order to analyse the T changes behaviour of the five different clusters (Figure 9). The effect of active thermography is clearly visible from the graphs. Indeed, independently from the class, the decrease/increase rate was considerably higher in the first 10 acquisitions when the fan was on. After the 10th acquisition, when the ventilation phase ended, temperature started decreasing/increasing slowly.

Looking at the classification results, it was possible to interpret and labelling classes. Five clusters were identified, from “Class 1” to “Class 5”. Wetter surfaces are those that reacted to ventilation with the highest decrease of superficial temperature. Those areas are identified by “Class 1”. “Class 5” indicates the relative drier area, where the fan action caused the lower decrease of temperature or, in some areas, it even caused an increase of superficial temperature drying the wall. Considering all zones, moving from the bottom (Zone 1) to the top (Zone 3) of the masonry, the effects of ventilation led to increased drying of the surface. Zone 1 was the wettest zone in which the fan action caused a decrease of temperature all over the surface. In Zone 2, pixels clustered in “Class 4 and 5” increased their superficial temperature due to ventilation by the 10th acquisition. In Zone 3, only “Class 1” could not reach an evaporative flux that allowed the surface to dry by the end of the experiment.

To take into consideration spatial temperature variation over the wall during ventilation, horizontal and vertical transect were analysed to look for defects (Figure 10). For each zone, the IR images T(x, y, 1), T(x, y, 5) and T(x, y, 10) were considered. From a temporal point of view, this analysis confirmed the previously discussed results. Zone 1 was the wettest area, and the area investigated by both horizontal and vertical transects registered the highest decrease of temperature between the start and the end of ventilation. From a spatial point of view, in Zone 1 the temperature decrease rate was more linear and homogeneous than in the others. T decreased from the left to right hand side of the masonry and from the top to bottom, as can be seen in the vertical transects. The analysis revealed that in Zone 2 and 3 the surfaces investigated were more heterogeneous. Defects and moisture were visible as negative picks over the transects, while drier surfaces had positive picks.

The different spatial behaviour of T change over surface can be related with the geometrics feature extracted from the 3D survey. Indeed, thanks to that survey, it was possible to recognize surfaces with positive of negative z coordinates with respect to the calculated best fit plane. In general, prominent surfaces are those characterized by better evaporative flux performance, and they can easily be dried by air jets. It is important to consider that some bricks were fully covered by moss, which can show prominently. Although these surfaces were prominent, they reacted to ventilation with a decrease of temperature and produced more homogeneous behaviour on the surface. An example is Zone 1, that is almost fully covered by moss.

4. Discussion and Conclusions

The presence of humidity is one of the major problems of degradation in buildings, especially in historic sites. This is the case of the Complex of Santa Croce, a UNESCO site since 1996. The site is located in a low-lying area affected by subsidence, and the floor of the Church, which is about 1.30 m above the sea level, is often flooded [7]. Due to its structural characteristics, the Church in the Complex is strongly affected by this form of degradation. The use of new non-invasive survey methods for the study of material conditions is needed for cultural heritage. In this context, infra-red thermography is a non-destructive techniques that has been successfully used for defects detection in materials and buildings [13]. Moreover, multitemporal passive thermography has been demonstrated to better capture and interpret dynamic thermal behaviour of buildings, such as using a time lapse approach [12]. Although this methodology can be very successful when climatic conditions change over time, active thermography is necessary in those cases where a temperature contrast is hard to recognize (Figure 2). The interior of Santa Croce Church, for example, has a relative humidity of about 80%, which makes thermal gradients over the walls very difficult to recognize with passive thermography.

In the current study an easy to reproduce methodology to acquire and process multitemporal thermal images to detect defects on very moist masonry was developed and tested in cultural heritage context. The use of a fan in the active thermography survey allowed study of the reaction of the masonry to air jets, identifying areas where humidity was more persistent. The multitemporal analysis of the IR images combined with clustering allowed meaningful information to be collected from the data. Indeed, K-means classification applied to the PCs made it possible to automatically detect surfaces characterized by different evaporative fluxes. Unsupervised classification did not require a priori knowledge about masonry surface condition, but clustering revealed the wettest surfaces. By selecting some points of interest from each class it was possible to plot the observed temperature over time and compare different behaviours. For examples, the surfaces with the highest water content were those where ventilation caused the highest decrease of superficial temperature. Moreover, transect analysis allowed comparison of temperature changes through space and time.

In conclusion, the main findings of the research can be summarized as follows.

• It was demonstrated that active thermography is able to highlight the different behaviours and different moisture contents of a wall in cases of extreme microclimates, i.e., almost constant temperature with very low gradient and very high humidity.
• The multitemporal approach to active thermography imagery helped to identify areas with different evaporative fronts, which were considered in the study as proxies for different damage levels. Moreover, it was possible to evaluate the temperature changes on the wall in both space and time.
• Image processing and analysis were performed with an unsupervised clustering technique that was able to automatically detect five classes of damage. As a result, a degradation map was produced.
• Such informative maps can be very useful as tools to assess the presence of moisture in the masonry, especially in the context of cultural heritage, allowing fast and precise remediation actions to preserve and recover wall integrity where it is needed most.

This study was part of a multidisciplinary work within the European H2020 SHELTER project. The results obtained can be integrated and correlated with those achieved by other research groups (Materials Engineering, Structural Engineering, Hydraulics, Geotechnics, Restoration, etc.) for meaningful interpretation and deeper understanding. Indeed, the variation of humidity in some areas could be correlated with the presence of voids in the mortar, efflorescence or other substances detectable through chemical analysis. It will be certainly useful in this sense to evaluate the collection and analysis of wall samples carried out by the Materials Engineering group in order to validate the procedure and improve the classification, Moreover, it should be determined if active thermography can detect temperature differences or different reactions to the ventilator due to biochemical factors detected by the lab analysis. The data obtained by thermal surveys could be included within the Building Information Model of the Church, generated by the same Geomatics group, to support the creation of a three-dimensional map of degradation.

The procedure was specifically developed for particularly humid conditions and tested on the presented case study. Future development should consider applying the presented methodology to other case studies including different cultural heritage sites with different masonry characteristics and different micro-climate conditions.