1. Introduction
Over the last few years, thermography has gained numerous applications in terms of studying the energy situation in the world. Since the industrial revolution, a large part of the world’s population has shifted to urban locations. As a result, the built environment expanded quickly, producing significant carbon dioxide emissions from the energy used in buildings. Therefore, energy consumption by buildings has become a widely spoken topic. The scientific community has grown interested in infrared radiation (IR) thermography [
1], especially because it is cost-effective and can provide images of different building elements representing the surface temperature. Buildings’ thermal radiance is recorded using thermal infrared (TIR) cameras, a time-efficient and economical tool for monitoring leakage and energy usage. It is also a non-contact and non-destructive method for thermal analysis. Thermal analysis [
2] is necessary for detecting patterns such as the inhomogeneous distribution of wall materials, anomalies, heating system pipes, internal cracks, structural damages [
3], leakages, moisture [
4], failure of electrical circuits, air conditioning, ventilation, etc., which are typically hidden from the building’s surface. Traditional methods of monitoring the thermal signature are manually looking at a series of TIR images captured from different viewing positions. Some drawbacks of directly using the images are that they include occluded objects and also require a significant overlap between multi-view images. Also, there can be temporal and illumination changes in transition areas between image acquisitions. Therefore, including a three-dimensional (3D) spatial reference containing thermal information in the building model can increase the scope of thermal building inspection [
5]. For complex buildings, instead of looking at isolated facades, the whole structure can be observed as a single unit. We can locate the anomalies with a 3D reference to the building and perform change detection and observation of dynamic processes more efficiently. Information from the inside of walls can be connected to the outside to identify energy loss.
Now, there can be different ways to represent 3D information, such as adding thermal textures to 3D geometric building models [
6,
7] or representing the thermal information in the form of voxels or meshes [
8] or thermal point clouds [
9,
10]. Some researchers in the past have devoted themselves to generating some of these kinds of representations. One thing to remember while using archived building models is that we can update the models from time to time, and acquiring the latest representation of buildings can be labor-intensive and time-consuming. Moreover, the levels of detail (LoDs) imply the precise geometric depiction of the building model. Also, the bigger vision should be to scale-up thermal models to vast areas like cities, states, or even countries.
The goal of our project is to enrich building models with thermal information. Hence, we have worked on generating textures by projecting point clouds to facades. Otherwise, we could have directly used 3D thermal point clouds. Also, later, for thermal analysis, we will use Neural Networks for image interpretation in the thermal textures. The interpretation of images is also more convenient than point clouds. Additionally, point clouds also contain occlusions, outliers, noise, and unwanted data. Some of this information is irrelevant and we do not want to process these extra information. Textures do not include this unwanted information, but only contain the information relevant to a particular facade structure.
This paper contributes a novel technique to texture facades of 3D building models using thermal point clouds. We assume that Mobile Laser Scanning (MLS) point clouds and thermal images were recorded in one measurement campaign and are fused to a thermally enriched 3D point cloud. We further assume that the point cloud was already used to refine the original building model by adding elements like windows and doors. A mapping algorithm takes the texture layer of the facade. For each texture pixel (texel), the algorithm finds the nearest optimal point closest to the vertical of the facade and which has a temperature value. A resolution layer weights the thermal intensities of all optimal points before assigning them to the texture layer. The generated TIR texture is enhanced and finally applied to the facade. This paper is structured into different parts.
Section 1 introduces the overall problem statement,
Section 2 discusses the related research conducted previously,
Section 3 describes in detail the methodology workflow,
Section 4 describes the test datasets and input data preparation,
Section 5 presents the implementation results and analysis, and
Section 6 finally draws conclusions and introduces future work.
In our mapping algorithm, we can compute the texel intensity using different methods such as nearest neighbor point, bilinear interpolation, cubic spline interpolation, etc. The primary reason that we have selected the nearest neighbor point is to keep the original temperature values. Nearest neighbor ensures the original temperature but at a slightly shifted position (lower geometric accuracy). On the other hand, interpolation has an estimated (weighted mean) temperature, but higher geometric accuracy (no geometric shift). An interpolation would have generated intermediate average values via some weighting criteria, which would have changed the original values. Another advantage of using the nearest neighbor is its simplicity, as we take the nearest point based on a distance metric without further calculating average values based on some complicated rules. But the downside of using the nearest neighbor approach is that some texels with no corresponding vertical neighbor points are left empty, so the generated texture image has some blank texels. Interpolation would have filled up all those empty texels. Also, with interpolation, we can further increase the resolution to obtain finer texels, but with the nearest neighbor, this will result in more blank texels. Hence, there is a trade-off between selecting the method that provides the real temperature at a slightly shifted position versus the method that provides an estimated/interpolated temperature with no shift.
The main innovation in this paper is the usage of thermal point clouds to generate thermal textures. Previously, researchers worked on generating thermal textures directly from TIR images or point clouds reconstructed from TIR images. Some classical texture generation methods project facade images directly onto the 3D building models. Due to the limited level of detail of these simplified building models, projection errors occur. Some other drawbacks, such as limited geometric density and lower resolution/accuracy of TIR images, lead to poor 3D reconstruction or blurry textures. Also, TIR images include reflections from window glasses and occlusions. Objects in the TIR domain show radiometric behavior, which causes blurred edges in the generated textures. In
Section 2, we discussed the drawbacks of facade texture generation from TIR images. Therefore, to overcome the limitations of these methods, we are using thermal point clouds prepared by extending laser scanner point clouds with intensities from TIR images. The reason why we used point clouds instead of images for texture generation is because of TIR image interpretation challenges, discussed in detail in
Section 2. We projected the thermal point clouds to building facades by using a mapping algorithm. The mapping algorithm uses a nearest neighbor approach. Later, we also implemented bilinear interpolation and compared that with the nearest neighbor. We also investigated different approaches for the nearest neighbor search to calculate the intensities, compared and analyzed them, deduced the best-suited approach, and finally presented a performance metric of the results.
2. Related Work
Some research has been conducted on preparing 3D thermal point clouds from different sources. Ref. [
11] reconstructed 3D point clouds containing TIR attributes using Structure from Motion (SfM) techniques with multi-view stereo-principle. The images were taken from Unmanned Aerial Vehicles (UAVs). Low-quality thermal images are challenging for this method since there must be enough detected features, especially when an uncooled camera captures the image. Ref. [
12] proposes an automated approach to register thermal with red, green, and blue (RGB) point clouds reconstructed independently via SfM. Normalizing the point cloud scale is the first step in the registration procedure. Global registration using calibration data and SfM output is followed by fine registration using an iterative closest point (ICP) optimization variant. Ref. [
13] generated dense point clouds directly from TIR images by automatically orienting sequences of images taken from UAVs. However, if RGB images are additionally available, dense point clouds are generated from RGB data. Then, the ICP algorithm is used to align the point clouds, a mesh is generated from the RGB point cloud, and, finally, a 3D model is created by mapping the texture from TIR images. Ref. [
14] superimposes thermal information onto each point on 3D structures reconstructed with Direct Sparse Odometry (DSO) using RGB images to create a 3D thermal map. The RGB and TIR cameras are mounted on a stereo rig and their relative pose is estimated. Depth images from RGB and TIR cameras are matched based on mutual information. Then, the point cloud’s scale is estimated corresponding to extrinsic parameters between both cameras to perform the superimposition. Ref. [
15] generated point clouds using SfM using cell phone images. Then, the temperature information was projected to the generated point clouds by utilizing a thermal camera’s relative position data. Ref. [
16] used a “bi-camera” system to integrate RGB and IR cameras and then performed thermal 3D mapping by registering the images into a terrestrial laser scanner (TLS) reference system.
Texturing building facades with thermal images was attempted by [
17]. Here, 3D models generated from terrestrial laser scanners are taken, and then RGB and TIR image blocks are mapped to the model. It uses accurate photogrammetric orientation via bundle adjustment that integrates TIR and RGB data. Ref. [
18] creates a 3D thermal map of a building using three sensors: a color camera, a 3D laser scanner, and a thermal camera setup in a robot. The sensors are automatically co-calibrated, and one highly accurate 3D model incorporating data from all the sensors is created to depict the heat distribution. Ref. [
19] maps roofs by fusing thermal and visible point clouds. The point clouds are generated from their respective thermal and visible UAV images. The point clouds are then geo-referenced using control points and are co-registered. Building roofs are extracted from visible point clouds and thermal point clouds, which are combined to create a fused dense point cloud. Ref. [
20] merges 3D point clouds with terrestrial thermal image data to map thermal attributes. Thermal and RGB point clouds are generated, respectively, from their images using SfM, and coarse registration is conducted between the point clouds. The RGB–thermal image pairs with the best correspondences are obtained, and reliable matching features are extracted from the pairs. In the end, the RGB image is mono-plotted to carry out fine registration, and then the thermal image is resectioned.
Previously, researchers have attempted to extract thermal textures for 3D building models. Ref. [
6] presents a mobile thermal mapping technique that extracts thermal textures using thermal image sequences, 3D building models, and point clouds. They introduced two workflows. The first workflow determines image orientations by co-registering terrestrial image sequences with a 3D building model. In the second workflow, given pre-orientations of TIR and RGB image-based point clouds are used to match them, and then the ICP algorithm is performed to minimize the distance between the point clouds. Finally, thermal textures are extracted by correcting the orientation parameters of TIR images. Refs. [
21,
22,
23,
24] refer to the concept of projecting the image sequence to the facade plane in a bundle adjustment process of tracked feature points to generate the facade’s thermal textures.
Figure 1 shows the facades textured from TIR image sequences based on these approaches. One drawback is that, compared with images in the visual domain, 3D reconstruction of TIR images has less accuracy due to the lower resolution of TIR images. It gives rise to more discretization and decreases matches between the image sequence. Another drawback is that objects in the TIR domain show radiometric behavior, which causes blurred edges and fewer changes in intensity and details in an image. Other drawbacks of the textures extracted using the above-discussed methods are that occlusions are not considered, and they also include window glasses, which often show the wrong temperature due to reflections, as illustrated in
Figure 1.
The terrestrial images cannot capture data such as roof structures, inner yards, or backyards. To complement this missing information, oblique images captured from a helicopter or aerial system are necessary. Ref. [
25] projects building models into directly georeferenced oblique airborne IR image sequences to perform texture mapping of the building models. Georeferencing, however, is not always precise; in this case, there is no match between the structures in the image and the projected model. Ref. [
26] solves this using line-based matching to correct the exterior orientation and determine the best fit between the image sequence and the 3D model.
Contrary to the texture representations, ref. [
9] attempted to represent the thermal information in the form of point clouds by preparing thermal point clouds through a combination of TIR image sequences and MLS point clouds. They combined data from various sensors using a data fusion technique. Firstly, they extracted key points from distorted TIR images, then used a restricted Random Sample Consensus (RANSAC)-based algorithm for correspondence determination to estimate the six Degree of Freedom (DOF) pose, and finally fused geometric properties from point clouds and thermal attributes from images using a non-local means strategy. Ref. [
27] attempted indoor mapping by adding thermal channels to 3D points by fusing TIR camera images with terrestrial laser scanners and RGB point clouds. They mounted the sensors on a robot platform and calibrated the system geometrically by defining the used coordinate systems and co-registering the laser scanner point cloud and camera images. As the sensors’ field of view differed, they had to synchronize them before fusion. A similar technique for co-registration with a photogrammetric point cloud is shown in [
28].
Ref. [
29] worked on IR camera calibration to map textures to 3D building models automatically. Texture mapping requires external orientation of images, and direct geo-referencing requires a global positioning system (GPS)/inertial navigation system (INS), which is included in the camera system that was calibrated. They used a helicopter platform to obtain the images. Ref. [
30] proposed a technique for texture mapping by finding the best fit between the airborne images and the building model using model-to-image matching. It also extracts texture by selecting the best texture. Ref. [
31] assessed the quality of building textures extracted from TIR image sequences from aerial and terrestrial platforms captured using various sensors and orientations. The qualities are compared based on the completeness of texture, viewing angle, the accuracy of the projection, and geometric resolution, to finally select the most accurate and complete textures. Ref. [
32] matched IR images with 3D building models for extracting textures. Based on the relative orientation of the point cloud, they matched terrestrial image sequences, and using standardized masked correlation, including system calibration, they also matched airborne images. They introduced the combination of airborne and terrestrial data in a single model. Ref. [
33] aligned 3D building models using oblique TIR images captured from a flying platform to extract thermal textures correctly. By establishing correspondences between image line segments and building model edges, they were able to track lines and estimate the optimal camera pose for the best possible match between the image structures and the projected model.
There are some benefits of using point clouds instead of images for texture generation. Images have drawbacks, such as that they include occluded objects and require large overlap between multi-view images. Another drawback is that objects in the TIR domain show radiometric behavior, which causes blurred edges in the generated textures and fewer changes in intensity and details in an image. They also include window glasses, which often show the wrong temperature due to reflections. Projecting or aligning the 3D building models with oblique images requires proper correspondences between the model and structures in the image to estimate the optimal camera pose for the best possible match, which is challenging and also needs accurate georeferencing and an accurate estimation of the external orientation of images. The number of elements visible in the TIR domain influences the position refinement. The refinement quality is greatly reduced in the case of facades with repetitive patterns or few objects. It becomes worse when it happens along with inaccurate GPS positions. The extraction of textures is sensitive to errors in the estimation of viewing direction. To overcome the aforementioned image interpretation challenges in TIR images, we used thermal point clouds. Now, thermal point clouds can also be generated only from TIR images via 3D reconstruction with SfM/the multi-view stereo principle. Still, the reconstruction accuracy will be limited due to the limited geometric density and lower resolution of TIR images that gives rise to more discretization and decreases matches between the image sequence. Homologous points are rarer to find and difficult to locate. Therefore, in this paper, we used point clouds generated with mobile laser scanning that give good location information with usable georeferenced 3D coordinates of buildings, good point density, and accurate geometric details, and then the points were extended with thermal intensities extracted from TIR image sequences. Also, when using point clouds, the occlusions are omitted, as our proposed algorithm selects only the nearest points from the facade for mapping.
5. Results and Discussion
5.1. Thermal Mapping
The computation and programming of all the processes discussed in this section were performed in MATLAB. The texture layer of one facade is taken with the texel size set to 10 cm × 10 cm corresponding to the real world (i.e., 10 cm GSD). The thermal point cloud is taken and then clipped to contain only points within a threshold 1 m in front and behind the facade, in the normal direction. As mentioned before, this reduces the computation time of the nearest neighbor search algorithm.
Figure 9 shows the point cloud before and after clipping, together with the facade plane. The facade is shown in red color and the point clouds in blue/yellow/green colors. From the figure, it can be seen that only the closest and relevant points within 1 m normal to the facade are retained after clipping.
The next step is to define the neighborhood for each texel, i.e., to find neighbor points in the thermal point cloud within a predefined radius from the mid-point of the texel. The radius is set to 1 m. Later, in
Section 5.3, it will be discussed how the performance is affected when we change the radius from 0.3 m to 1 m.
Figure 10 shows the neighbor points (green) for a single texel plane (red).
Then, the algorithm searches for the nearest point vertical to the texel and which has a temperature attribute. Three methods are approached: angle to normal vector, perpendicular distance to normal vector, and distance without considering their verticality. We already discussed these three approaches in detail in the previous section. Their performance results will be discussed and compared later in
Section 5.2.
Figure 11 shows the detected optimal point closest to the vertical (blue) for a single texel plane (red), the nearest distance point (green), the farthest point within a predefined radius (red), and lines connecting the texel mid-point (black) to the detected points for illustration. The figure shows both the top view and side view. From the figure, it can be clearly seen that the nearest distance point is not normal to the texel mid-point, and the point closest to vertical is a bit far from the nearest distance point.
For illustration,
Figure 12 shows how the detected points look for the whole facade plane (red), considering texels at a distance 3 m apart from each other and skipping all the other texels in between them. It can be clearly seen from the figure that some detected optimal points (blue) are behind the facade. This is obvious because the mapping algorithm searches for points on both sides of the facade.
After finding the optimal point, the thermal attribute of that point, weighted by the resolution layer, is mapped to the texel.
Figure 13a shows the resolution layer applied to the texels, with the brighter pixels having better quality and darker pixels showing worse quality. From the figure, it is clear that all the points are not of the same quality and, especially, more points on the right side are bad quality. Therefore, it makes complete sense to weight the thermal intensities with quality information before mapping. The upper right corner has completely black pixels, as it does not have thermal attributes. And this is evident when we look at the input point clouds in
Figure 8, as they have no points in this corner region. The mapped thermal texture is a low-light image that is brightened and a heat map is applied as a color map to visualize the temperature distribution better.
Figure 13b shows the resulting generated texture. In the texture, we can see some heating pipes in the shape of cylinders; thermal lintels, bridges, and anomalies in the shape of blobs (near windows/doors).
Finally, the extracted gray texture mask, shown in
Figure 7b, is applied to remove the doors and windows and to assign a uniform gray value to the texels where no thermal intensity is mapped. The mask is binarized via thresholding and blended with the texture in
Figure 13b to generate the final TIR texture, as shown in
Figure 14a. It is implemented on two facades, as shown in
Figure 14b.
The source thermal point cloud shown in
Figure 8b and the resulting textured facade in
Figure 14b look quite consistent. Therefore, this proves that the projection algorithm works quite well. Also, in comparison,
Figure 14b clearly shows that the occlusions are not mapped, as the algorithm selects the nearest points from the facade for mapping, unlike in the case in
Figure 1. Additionally, the windows are excluded by using texture masks. Therefore, our method shows an advantage over the previous approach.
5.2. Comparison of Different Approaches for Searching Nearest Point
Earlier, we discussed three different approaches for searching for the nearest point. In this section, we will compare their performance results. The purpose of this analysis is to find out the best-suited approach that yields the best texture resolution at the best computation cost. The first approach minimizes the angle between the vector normal to the texel and the vector connecting the optimal candidate point to the texel mid-point. The second approach minimizes the 3D perpendicular distance between the optimal candidate point and the normal vector. The third approach is to select only the minimum distance point with a temperature value without considering if it is vertical to the texel or not. The second approach yielded the finest pixels.
Table 1 shows the detection results based on the three approaches for one facade.
The computation time is apparently much more for the first two approaches because it has to complete the extra computation to find the point closest to vertical. On the other hand, the third approach is quite fast, as it only searches for the nearest point. Also, it has the least mean distance from the optimal points to texels, which is quite apparent. As it does not consider verticality, it has a high mean angle and high mean perpendicular distance to the normal vector. The mean angle to normal for the first approach is 7.19°, which is very close to normal and looks good. For the second approach, the mean perpendicular distance to normal is 0.0413 m, which is also close to vertical and is quite satisfactory. Considering the above factors, there is a trade-off between the texture resolution and computation time. The second approach yields the finest texture, but at the cost of processing time (186 min), while the third approach generates the coarsest texture, but is quite fast (23 min). Therefore, the user can choose one of these approaches based on their needs and priorities.
5.3. Tuning the Radius to Find Neighbor Points of the Texel
As mentioned earlier, to define the neighborhood of the texel, we need to set a predefined radius. The radius signifies the search space for the nearest neighbor. The larger the radius, the more the search space, and vice versa. The selection of an optimal search space is crucial for the thermal mapping algorithm. Therefore, the purpose of this analysis is to find out the deciding factors for an optimal search space. The mapping results are affected by changing the radius, which is discussed in this section. We want to analyze what is the best-fitting radius that yields a good detection rate and a good quality texture in a reasonable processing time.
Table 2 shows the results for one facade.
The detection rate signifies the proportion of texels for which a corresponding temperature value can be detected and mapped. With an increasing radius, this rate increases, which seems logical because the search space increases as more neighbor points are considered. The computation time also increases as more points need to be processed. The mean distance from the optimal points to texels increases, which is also obvious because more points imply more outliers. The mean perpendicular distance to the normal vector is approximately 0.08 m, which is close to vertical. Therefore, it looks like the deciding factors are the detection rate and computation time. Changing the radius from 0.3 m to 0.5 m increases the detection rate significantly. It can also be seen in the generated textures, whose quality improves (becomes finer) and has fewer texels with no thermal intensity. But the computation time also increases three-fold. Changing the radius further to 0.7 m or 1 m does not improve the detection rate much, but the processing time keeps increasing significantly. Therefore, looking at these factors, a radius of 0.5 m seems to be a good trade-off where one can assure good quality in a reasonable processing time. Moreover, the best dimensions of the neighborhood will also depend on the texel size and point cloud density. If the GSD is reduced, fewer neighbor points will be available for each texel and, therefore, need a larger radius to include more points. Similarly, more points are available per texel for a higher point cloud density, so a lower radius would suffice. Nevertheless, reducing the GSD and increasing the point cloud density will definitely require more computation cost and time. Therefore, considering these factors, an automated method to deduce the best-fitting dimensions of the neighborhood will be quite useful.
5.4. Bilinear Interpolation
As an additional experiment, we computed the texel intensity using bilinear interpolation. To validate our choice of the nearest neighbor approach, we compared it with bilinear interpolation. Later, in
Section 5.5, we perform an accuracy analysis of these two methods and show that the nearest neighbor approach outperforms the bilinear interpolation.
For bilinear interpolation, we project each point in the point cloud normally to a facade plane. Then, we take the interpolated value of four projected neighbor points for each texel. The bilinear texture mapping process can be described by the following steps:
- (i)
Filter and take planar points from the point cloud. Fit a plane to the thermal point cloud with a maximum allowable distance of 0.5 m from the facade. Retain all the inlier points in the plane and discard the remaining points in the point cloud. We used the M-estimator SAmple Consensus (MSAC) algorithm to find the plane.
Figure 9b shows the planar points.
- (ii)
Project each point in the plane normally to the facade. We only projected the points with a temperature value. Therefore, the points with no thermal attributes are discarded.
Figure 15a shows some sample points (blue) from the point cloud projected normally to the facade plane. The points after the projection are depicted in green.
- (iii)
Take the texture layer of the facade (with the same GSD 10 cm). We take the mid-point of each texel and find its four nearest neighbor points from the above-projected points. The nearest neighbors are found by using the Kd-tree-based search algorithm. In
Figure 15b, we show the mid-point of the texel in blue, the four nearest neighbor points (
) in green, and the corresponding Euclidean distances (
) between the mid-point and the neighbor points in yellow.
- (iv)
Finally, for each texel, we perform bilinear interpolation of the thermal attributes of the four neighbor points.The interpolated value is based on a weighted mean, with the highest weight given to the lowest distance point and vice versa. The interpolated value is calculated as:
where
are the weights and
and
are the temperature values (weighted by the resolution layer) corresponding to the neighbor points
and
. The highest weight,
, corresponds to the nearest point,
, and the lowest weight,
, corresponds to the farthest point,
. Therefore,
.
The interpolated thermal value is finally assigned to the mid-point of the texel. We repeated this step for all texels until all the interpolated values were mapped to the facade.
Figure 15c shows the thermal texture generated by bilinear interpolation. The interpolation changed the original temperature values and generated some intermediate average values, so all the texels in the generated texture image are assigned a value and there are no blank texels. Therefore, we can see that the upper right corner is also filled up with some interpolated values, even though there are no corresponding points in that region in the input thermal point cloud (
Figure 8).
5.5. Performance Evaluation
In our study, we registered point clouds with 3D building models. The registration may cause some errors, such as shift and tilted point clouds. This can lead to incorrect projection directions and can impact the mapping results. For example, the mapped temperature intensity of a point may shift from its original location. Therefore, it is necessary to evaluate the mapping accuracy. The purpose of this analysis is to evaluate and compare the mapping accuracy of our proposed method and some similar/former methods.
The accuracy or correctness of the texture is evaluated based on the window’s corner points. We manually selected some corner points which were distinctly visible in the thermal texture. We calculated the deviation/shift of these points from a ground truth RGB texture of the facade. From the discussion in previous sections, the approach “Minimize perpendicular distance to normal” for searching the nearest point yields the finest texture, and a radius of 0.5 m seems to be a good trade-off to obtain good quality at a higher detection rate. Therefore, we took these as a use case for calculating the performance metrics.
Figure 16a shows the window corner points (blue) in the thermal texture and
Figure 16d shows the ground truth RGB texture (corners marked in green).
For deviation, we calculated the Euclidean distance between the pixel coordinates of the generated thermal texture and the ground truth RGB texture. Finally, we evaluated the root-mean-square deviation (
RMSD) of our nearest neighbor approach as follows:
where
and
are the corner point coordinates of thermal texture (nearest neighbor),
and
are the corner point coordinates of ground truth RGB texture, and
N is the total number of corner points.
We obtained an value of 0.7 m, which seems satisfactory. For perfect results, this value should tend to zero.
To further strengthen the accuracy analysis of our proposed method, we will compare our study’s results with similar/former studies. As a use case for similar study, we take the thermal texture generated using bilinear interpolation (shown in
Figure 16b, corners marked in black color) and calculate its deviation from the ground truth RGB texture. The
RMSD value for bilinear interpolation is calculated as:
where
and
are the corner point coordinates of thermal texture (bilinear interpolation).
The estimated value is 1.6 m, which is more than the value for nearest neighbor. This proves that the nearest neighbor has better accuracy than bilinear interpolation.
As a use case for previous studies, we take the texture generated from a sequence of thermal infrared images [
6,
24]. This is shown in
Figure 16c (corners marked in green). Similarly, we calculate its deviation from the ground truth RGB texture. The
RMSD value for the image texture is defined as:
where
and
are the corner point coordinates of the texture from TIR image sequences.
The value is estimated to be 1.37 m, which is again more than the value for nearest neighbor. This shows that the nearest neighbor also has better accuracy than the texture generated from TIR image sequences.
Therefore, from the accuracy point of view, our proposed nearest neighbor method outperformed both bilinear interpolation and texture from image sequences ().
Therefore, the performance evaluation supported our choice of the nearest neighbor approach over bilinear interpolation. Earlier, we mentioned that texture generation from TIR images can lead to projection errors due to the limited level of detail of building models. We verified that since our nearest neighbor method is better in terms of accuracy (
). Our approach does not map occlusions, which is evident in
Figure 1 and
Figure 13b. Therefore, our results verify our contributions and the gaps we fill in the literature.