Material Inspection of Historical Built Heritage with Multi-Band Images: A Case Study of the Serranos Towers in Valencia

Abstract

Built heritage materials assessment is an important task for planning and managing future conservation works. The uniqueness of each historical building makes reconnaissance operations more complex and specific for every single building. In the past, visual inspection and invasive techniques were widely used to investigate surface materials. Non-destructive techniques (NDTs) such as multi-band photogrammetry and remote sensing can help to assess the buildings without any contact with the investigated objects, restricting the disruptive tests on limited areas and reducing the testing time and costs of the surveys. This paper presents the results obtained using multi-band images acquired with a low-cost imaging solution after interchanging several filters, and the application of the principal components analysis (PCA) to recognize different materials of a significant historical monument. The Serranos Towers, built between 1392 and 1398, suffered several interventions in the past that affected their state of conservation with the replacement of different materials. The results of the study show the usefulness of applying PCA to distinguish different surface materials, often similar to the original ones, in a fast and efficient way to investigate and analyze our heritage legacy.

1. Introduction

The conservation of historical built heritage is a process that involves a deep understanding of building, not only related to materials but also damages, conservation policy, diagnoses, and selection of ideal treatment to safeguard monuments for future generations. The knowledge and the documentation of the construction materials is a fundamental and preliminary task for every consecutive phase of restoration and transformation [1]. The uniqueness of every building, its history in intervention, and reuse in the past, makes it more difficult to perform proper reconnaissance operations due to the singularity of the building [2].

In the past, material recognition was performed by experts with invasive techniques [3,4,5], often localized in small areas, being previously investigated based on visual inspections. Thus, the development of alternative methods, such as non-destructive techniques (NDTs), that can help to investigate materials and damages is a topic of great interest to reduce destructive tests in limited areas and to reduce time-intensive and usually expensive surveys. Among the NDTs, digital image processing, photogrammetry, and remote sensing are very suitable for this aim due also to their capability to acquire information without contacting the investigated building. Imaging sensors record the reflectance response of an object in different ranges of the electromagnetic spectrum (Near-UltraViolet (NUV), visible (VIS), near-infrared (NIR),InfraRed Thermography (IRT), etc.), with or without the fusion of 3D laser scanning data [6,7]. The analysis of the spectral response of objects allows the discerning of various materials, components, and damages [8,9,10].

Concerning the electromagnetic spectrum investigated, several techniques for material inspection are available, ranging from conventional visible images [2,11], multi-spectral and hyper-spectral images [3,8,12,13,14], to thermography [15,16]. They can be used alone or integrated together [16,17,18] to improve the results. Other recent developments include the usage of LiDAR or photogrammetric techniques to combine the material analysis with the geometrical information [19,20,21,22,23] and disseminate the comprehensive results [24,25].

Remote-sensing techniques are widely used in other fields for material and object recognition, such as in archaeology [26,27] or fine arts [18,28,29]. However, little research is conducted on the topic of material surface recognition and mainly they are devoted to degradation and damage to the structures.

In the field of cultural heritage, non-invasive and micro-invasive techniques are now starting to be consolidated. In Martinho and Dionisio (2016) [30], a summary of these techniques is reported. In particular, the geophysical methods are devoted to assessing both qualitative and quantitative information on materials and deteriorations, using sonic and ultrasonic methods, ground penetration radar, and electrical methods [30]. The integration of several techniques allows users to obtain different characteristics and a more complete description of the investigated object. Specialized non-destructive and micro-invasive approaches typically used for cultural heritage diagnosis are expensive and localized in small areas. They are performed after an exhaustive expert’s visual investigation [31]. The latter is considered time-consuming and additional platforms for reaching the target areas are required. Therefore, the development of NDTs and, in particular, multi-band imaging methods such as the one presented in this paper can aid the visual and material inspection using low-cost tools, improving the efficiency and reliability of surveys.

Sanchez et al. (2017) [8] investigated the use of low-cost photogrammetric equipment to capture RGB and near-infrared (NIR) images for the automatic classification of materials (unaltered and eroded) in historic buildings. They compared various classification methods applied to both RGB and RGB + NIR images, concluding that the inclusion of NIR improves the accuracy of material recognition, with the best classification result being achieved with the maximum likelihood algorithm. Similarly, Zahiri et al. (2021) [21] reached a comparable conclusion by examining the combined use of multi-spectral imagery and LIDAR intensity data to identify and classify building materials such as concrete, brick, and mortar, as well as distinguish between different compositions (e.g., brick firing levels). In summary, multi-spectral images, incorporating NIR and RedEdge bands, achieved classification rates exceeding 80%, with or without the LIDAR data. Furthermore, Zahiri et al. (2022) [14] compared hyper-spectral and multi-spectral images for the detection of five different materials, namely brick, mortar, stone, plaster, and painted windows. The results confirm that hyper-spectral images work better than multi-spectral images (overall accuracy 80%) but, given the high cost of hyper-spectral sensors, in some contexts, multi-band sensors can be replaced instead. In fact, the combination of band-pass and long-pass spectral filters in the spectral range of NUV, VIS, and NIR allows users to consider any modified solid-state camera, i.e., digital camera without NIR cut-off filter, as a low-cost multi-band mapping sensor [8,29].

For proper interpretation of multi-band/multi-spectral/hyper-spectral data, several image processing algorithms can be applied, from classical applications of spectral [32,33] and spatial processing [34] to the new innovative machine learning, deep learning [35], etc. A classical and well-known approach is the principal component analysis (PCA) [36], a multivariate statistical data analysis method used in several fields, such as medicine, chemistry, economy, etc. In the field of remote sensing, as well as in the digital image processing world, the PCA is mainly used to summarize the overlapping radiometric information readable from the spectral bands. In particular, it is applied in hyper-spectral image dataset reduction to simplify the complexity of data [37], minimizing the number of spectral bands. Examples of its application can be differentiated in the literature: in multi-spectral and hyper-spectral images to improve classifications [37,38,39]; other interesting research concerns the usage in the field of museum objects [40,41], painting art [18,28,29,34,42,43] and ancient manuscripts [44,45] to analyze and reveal hidden features, not recognizable in the visible images.

In this paper, we perform a multi-band survey to extract material information and improve the recognition of different surface materials of a historical monument built in Valencia (Spain): Serranos Towers (‘Torres de Serranos’), which has undergone several interventions during the past. The Serranos Towers case study and the main intervention phases will be introduced in the next section. Later, the survey and processing method for surface material recognition will be presented. Finally, a discussion of the results and conclusions are reported.

2. Materials and Methods

This section is organized as follows: an introduction to the case study and the history of the investigated building (Section 2.1), the data acquisition phase and the equipment (Section 2.2), and the pre-processing phase (Section 2.3). Finally, PCA to detect materials is presented in Section 2.4.

The methodology followed to extract material information and its metric features is shown in the workflow (Figure 1).

2.1. Case Study

Serranos Towers, situated in the northwest of Valencia city (Spain), constitute one of the few gates that remain after the demolition of the medieval walls of the city (Figure 2). The building consists of two large crenelated towers, once connected to the city walls, and a lower central body, which allows, through a large semi-circular arch, the entrance to the city. Inside each tower, three naves are opened on three different levels, while the central body constitutes the connecting space. They were constructed by Pere Balaguer as part of the city’s fortification at the end of the 14th century (between 1392 and 1398). During the centuries, the Serranos Towers suffered several interventions [46]. In 1586, Serranos Towers were converted into a prison (1586–1872), so important interventions were carried out to adapt them to their new functional usage. Indeed, the Serranos Towers underwent major changes with respect to the original structure, such as the creation of windows on the façades (fourteen), sentry boxes, a moat closure, and so on. After 1872 (between the end of the 19th and the beginning of the 20th century), the most important restoration works were carried out to restore the towers to their initial configuration.

During the 20th century, other major interventions were performed to safeguard the integrity of the monument, replacing some excessively degraded elements such as keystones and ashlars. Besides, in the 21st century, the municipality of Valencia (Ayuntamiento de Valencia) commissioned further works of maintenance and conservation of the Serrano Towers [46,47]. During the 2000–2002 renovation works, pre-intervention studies were performed, including a photogrammetric survey to acquire geometric information following traditional analytical stereoscopic plottings, and several on-site and off-site material inspections and studies, such as historical investigations, stratigraphic, petrographic, and glyptographic studies, degradations, weather, and biodeterioration analyses, investigations on the materials of doors and locksmiths and their state of decay, polychromies of the decorations, georadar and subsurface core surveys, and cleaning tests [46]. Thanks to these existing studies, it will be possible to validate the PCA results presented in the next sections.

2.2. Multi-Band Survey

The survey is concerned with the north façade in the central body of the Serranos Towers (Figure 2). The north façade is the main façade of the Serranos Towers and has a magnificent tracery in the upper area and the arch that represented the entry to Valencia city center. This façade represents an area where several restoration interventions have been carried out over the years, and where many original elements have been replaced with similar ones.

A survey with a FujiFilm Camera IS PRO was conducted. The sensor captures the light spectrum ranging from UV-A or NUV (about 380 nm) to the NIR spectrum (around 1000 nm). The NIR cut-off filter, built into most digital cameras, is not manufactured in this particular SLR (single reflex camera) camera model; thus, there was no need to modify the SLR camera. Therefore, the camera allows us to register imagery in a wide spectrum range. However, some optical filters from MidOpt^® [48] were used to acquire the RGB, NIR, and NUV multi-band. To obtain the visible images (RGB), a bandpass filter BP550 (range 425–770 nm) was used to block the NUV and NIR radiation; for the NIR images, a BandPass BP800 (range 745–950 nm) was used. Finally, a double filter was combined (BP365 + SP730) to obtain NUV images: the BP365 filter allowed the acquisition of the range of wavelengths 335–400 nm and part of the NIR spectrum; to minimize (block) as much as possible the NIR spectrum, the shortpass SP730 was also mounted on the camera. The survey was carried out around 12:00 p.m. on 7 October 2020, on a slightly cloudy day and the temperature was around 26 °C. One image was acquired to cover the entire tracery and a portion of the arch below. The distance-object acquisition was about 15 m.

2.3. Pre-Processing

The acquired images (NUV, RGB, NIR) were elaborated with Hypercube software v11.52, a free software package released by Geospatial Research Laboratory (GRL) of the Engineer Research and Development Centre [49] (ERDC).

First of all, NIR and NUV images were acquired as three color band images, so conversion into grayscale was required and it was obtained using a gray equal combination. Moreover, NUV and NIR images presented a slight shift and rotation effect with respect to the visible ones, due to minor camera movements while capturing the different filtered images. A warping process was performed to align the NUV and NIR images with the visible ones. This procedure required the measurement of some tie points and imposing a 2D transformation. Several transformation models can be selected in the function of the images used. For our purpose, the orthogonal transformation that preserves shape, allowing only translations, rotations, and a single scale change, was chosen. A least-squares transformation was used to minimize the error across the image with root mean square errors (RMSE) smaller than one pixel, measuring 5/6 tie points for warping the images. During this process, the images underwent a bilinear resampling, although no significant differences were found with other interpolation methods such as nearest neighbor and cubic resampling. Once the images were geometrically corrected, a multi-band image combining NUV, R, G, B, and NIR images (Figure 3) was created, yielding a 5-band cube. To obtain metric information on the main façade of the central body, a projective rectification of the images with 6 tie points was performed, merging the cube images with elevation maps achieved by the photogrammetric reconstruction during the survey of 2000–2002. This geometric transformation was suitable because the façade is flat; otherwise, an orthorectification would have been required [50].

2.4. PC Analysis

After the initial pre-processing presented above, PCA was performed. The PCA allows transforming an original set of variables into another set of orthogonal variables or components, uncorrelated with each other, obtained by a linear combination of the above, so that the first generated component explains the maximum of the total variability, while the rest of the resulting orthogonal components are ordered based on the variation in the dataset [15]. In other words, this means finding new variables that maximize the variance and that are uncorrelated with each other. In general terms, PCA of an image are obtained according to:

$${\mathrm{PC}}_j=\sum _{i=1}^n{a}_{i,j}·{\mathrm{DL}}_i+R_j$$

(1)

where ${\mathrm{PC}}_j$ indicates the digital level (DL) of each pixel corresponding to the j-th principal component, $a_{i,j}$ is the coefficient applied to the DL corresponding to the i-th band to generate the j-th component, $R_j$ is a constant introduced to avoid negative values, and n is the number of input bands [15]. As a result, n principal components (PCs), equal to the number of input bands, are obtained. PCA was applied to the rectified cube of multi-band images, obtaining a new cube of 5 PCs. These new images are used to analyze and discern material changes in the Serranos Towers.

3. Results

The PCA was applied to the north façade of the Serranos Towers to analyze the tracery, the arch, and its surroundings. Five PCs (PC1, PC2, PC3, PC4, PC5) were obtained (Figure 4) from the 5 input bands (Figure 3). The last PCs, PC4 and PC5 in particular, gave interesting information about the construction materials in the investigated areas.

PC5 (Figure 5a) highlights the presence of the two main materials in the tracery. It is important to underline that tracery is a mixture of different materials, processes, and substitutions from different periods. According to the material map (Figure 5b), obtained during pre-intervention studies conducted in 2000–2002 [46], the tracery is realized with 2 main stones, calcareous tuffs (letters A/B in Figure 5b), and dendritic stones (letters C/D in Figure 5b). In particular, the calcareous tuff “A” is a porous stone and occasionally conserves the original red coloration, as it is possible to see in the visible image (Figure 5c). The “B stone”, meanwhile, is a compact calcareous tuff. The difference between them is the different stones’ finish techniques (A is a carved stone and B is bush hammering), as can be appreciated in the scatterplot presented in Figure 4f, in contrast to the scatterplot presented with the input images at both spectral ranges (Figure 3f).

The “C group” has different subgroups, numbered from 1 to 5, that represent a particular treatment of the stones. Finally, another class, D, was identified and included in the class of dendritic stones that are too degraded to be assigned to the previous categories. For a complete description of the material composition of the tracery, see the book [46]. PC5 (Figure 5a) differentiates the A/B calcareous tuff stones (red square in Figure 5b) with respect to the dendritic ones (C/D stones). As stated before, A and B stones are calcareous tuff, and the difference lies in the stone’s finish technique: A is a carved stone, while B is bush hammering. The same applies to the stones C and D. The clear distinction between the two main stones A/B and C/D is shown in the scatterplot (Figure 4f).

In addition, PC5 shows two stones (highlighted in green in Figure 5b), depicted as calcareous tuffs and identified as dendritic stones (C and D) in the material map. After a deep and detailed new investigation of the data acquired in 2000–2002, a materials expert established that the two stones are calcareous tuff, confirming the utility of the method to identify “human error” based only on the visual inspection. The black and dark gray on the PC5 (Figure 5a) represent stones with red pigmentation, highlighted in blue in Figure 5b. Probably in the past, all the tracery was covered with this red layer, and today it is visible only on the original stones [46].

In addition, the PC5 band in Figure 6a also showed part of the reconstructed arch below the tracery. Some ashlars were replaced during the recovery works at the end of the 19th century, as shown in the period construction hypothesis map realized in 2000–2002 and in the historical picture dated 1917 (Figure 6b,d). Today, the difference is not very visible to the naked eye, as shown in the visible image acquired during our survey (Figure 6c).

“Porous calcareous tuff” was the original stone used to build the arch. Rather, the stones that have been replaced are “compact calcareous tuff”, which is the type of stone that was primarily utilized for 19th century restoration. These two stones have similar spectral responses (Figure 7); therefore, it is very difficult to discern the differences based on the separability of the spectral signature only.

Finally, the PC4 band highlights new stones in the upper part of the arch. During the prison period (1586–1872), the emblem upper the arch was replaced by a window, the latter was removed during the third recovery phase with the replacement of new stones different from the adjacent ones (Figure 8). Also in this case, as stated in [46], the replaced stones are “compact calcareous tuff”. Similar results are obtained in the upper part of the tracery. Comparing the results with the period construction hypothesis map, it can be seen that, in some cases (green box in Figure 8a), the stones are partially recognized, because they are affected by some degradation forms that should be further investigated; only one replaced stone is not recognized (red box in Figure 8a), despite being distinguished in PC4 (Figure 4d).

4. Discussion

The application of PCA on the set of multi-band images allowed for highlighting the presence of different materials used in the interventions undertaken in the past. On one hand, it was able to identify different materials with different features (calcareous tuff and dendritic stones) and, on the other, similar materials with an analogous spectral signature were highlighted. The PCA is a standard multivariate statistical approach used in different contexts to study the variability of datasets. In the field of image processing and remote sensing, it is applied to improve the readability of images acquired in different spectral ranges (visible, NUV, NIR, TIR, etc.), and typical applications extend from archaeological and landscape analysis to museum objects, fine art, old paintings, etc. Regarding built cultural heritage, some PCA applications on thermal imagery, often applied to study decays and damages to materials, are conducted but few applications are shown on the use of PCA to analyze and improve the material reconnaissance using low-cost NDTs including hardware and software. Other research and investigation methods for material recognition focus on developing classification techniques based on multi-band/multi-spectral images [3,8,13,21,50], due to the separability of spectral signatures of materials. In some cases, as in the case of the arch (Figure 7), in which the original materials were replaced with similar ones, the classical supervised and unsupervised classifications following a pixel-based approach did not allow us to satisfactorily discern small differences featuring a similar spectral response.

The reconnaissance of materials is one of the preliminary tasks for every subsequent maintenance or intervention work in our built heritage. The classical procedures require an exhaustive on-site visual inspection, sample extraction for laboratory analysis, and detailed data analysis [4,5]. However, they are considered time-consuming, may require additional platforms for reaching the target areas, and some often apply controlled destructive techniques, so the development of auxiliary multi-band imaging techniques or alternative 3D surveys can easily complement and help experts’ analysis and understanding.

Regarding the methodology, the advantages of using non-destructive remote-sensing techniques are: in the first place, the differentiation of the materials even in cases in which there is a strong similarity, as has been shown in this study; in the second place, if the PCA methodology is applied before the visual inspection takes place, the differences can be highlighted and bring the attention of the experts. Worth noticing is that for the differences there is no need to carry out previous radiometric calibration of the imaging sensors, differently from other applications interested in measuring the physical object reflectance values [9,13,45]. These differences can be investigated with both visual inspection and petrographic studies in the case of stone-based surveys, or with other specific studies for other materials. In this sense, it is important to bear in mind that the PCA methodology does not allow the identification of the petrological nature of the material, but, as has been shown in this article, it shows differences linked to various factors (petrology, stone carving, presence of skids) that allows saving an important time in the visual inspection. At the same time, it can be used as an objective guide for establishing the samples that must be extracted to carry out micro-destructive studies.

Another advantage concerns the usage of low-cost sensors (any camera without NIR cut-off filter can be used) in combination with spectral filters, although this latter comment depends on the spectral signature of the constituent materials, and sometimes this is not possible with conventional solid-state (CMOS/CCD) sensors. Last, but not least, the illuminant spectral range is advantageous; if the survey is taken outdoors with sun daylight, this latter point is not an issue, contrary to indoor applications requiring different spectral range lighting [29,45].

Finally, the photogrammetric surveys can be easily enriched with additional spectral information that amplifies damages and/or differences, not easily identifiable on datasets using visible and/or thermal imagery.

5. Conclusions

This study has demonstrated the effectiveness of a low-cost NDT methodology that combines surveying and geomatics processing tools for recognizing materials in built heritage. By using a digital camera without an NIR cut-off filter and some low-cost spectral filters to acquire multi-band images, we successfully applied the classical principal component analysis (PCA) procedure to investigate and map different stone materials.

The case study of the Serranos Towers in Valencia, chosen for the availability of exhaustive historical and architectural studies, yielded interesting results in distinguishing various types of stone, often similar to the original ones. The PCA carried out has allowed us to confirm the matching of constituent materials, in good correspondence with previous pre-study interventions conducted in 2000–2002, when the conservation works were commissioned on the Serrano Towers.

These results are encouraging, and they can help technicians and professionals quickly identify material differences and conduct a more in-depth investigation of the areas identified with the methodology. Although the technique does not investigate the nature of materials, it can detect important information related to different factors, such as petrology or stone carving, etc., which allows saving important time in the classical visual inspection.

In the future, more in-depth analyses will be performed to improve the results, looking in particular at both the influence of treatments on materials and how degradation and damage can affect the results. They will be performed by integrating additional data sources, such as thermal data and laser scanner data based on other spectral ranges. It will help to improve the separability and recognition of materials and degradation. In addition, other image processing techniques that can improve the results will be tested.

The development of NTD methodologies is of great interest in the field of cultural heritage, as it allows the avoidance of invasive and micro-invasive tests for a better conservation of the asset itself and to develop more objective investigation techniques.