Automatic Detection of Linear Thermal Bridges from Infrared Thermal Images Using Neural Network

Abstract

Detecting thermal bridges in building envelopes should be a priority to improve the thermal performance of buildings. Recently, thermographic surveys are being used to detect thermal bridges. However, conventional methods of detecting thermal bridges from thermal images rely on the subjective judgment of audits. Research has been conducted to automatically detect thermal bridges from thermal images to improve problems caused by such subjective judgment, but most of these studies are still in the early stage. Therefore, this study proposes a linear thermal bridge detection method based on image processing and machine learning. The proposed method includes thermal anomaly area clustering, feature extraction, and an artificial-neural-network-based thermal bridge detection. The proposed method was validated by detecting the thermal bridges in actual buildings. As a result, the average precision, recall, and F-score were 89.29%, 87.29, and 87.63%, respectively.

1. Introduction

According to ISO 10211:2007, a thermal bridge is defined as “part of the building envelope where the otherwise uniform thermal resistance is significantly changed by full or partial penetration of the building envelope” [1]. Thermal bridges account for a significant proportion of the heating load in buildings [2]. Previous studies have reported that thermal bridges increase the heating load by up to 30% [3,4]. Thermal bridges not only increase the amount of energy consumed to cool and heat buildings, but also lower the thermal comfort of occupants because of cold outer walls [5] and cause serious problems, such as condensation [6]. As described above, thermal bridges in building envelopes have negative effects on the thermal performance of buildings, so detecting the location of these thermal bridges helps to develop a strategy to retrofit building envelopes [7]. Therefore, detecting thermal bridges in building envelopes should be a priority to improve the thermal performance of buildings. Thermographic surveys are recently being used as a useful method to detect thermal bridges in building envelopes [8]. Thermal anomalies, including thermal bridges in buildings, affect the surface temperature of the envelope. An infrared thermal imaging camera is a device that converts the radiation emitted by an object into temperature and presents it in the form of images [9]. The infrared thermal images acquired by a thermal imaging camera shows thermal anomalies caused by the surface temperature differences in the envelope, and an audit can visually detect thermal defects in building envelopes through these images [10,11]. In general, a thermographic survey proceeds as follows. First, the auditor moves around the building and uses a thermal imaging camera to check the temperature distribution of the building envelope. If the auditor finds areas with abnormal temperature distribution in the thermal images, the auditor analyzes these areas. During this analysis, the auditor generally uses experience and subjective judgment to detect thermal bridges [12,13]. Therefore, as these results are significantly influenced by the auditor’s experience and subjective judgment, the auditor may either identify or fail to detect actual thermal bridges.

Recently, several studies have been performed to automatically detect thermal bridges from thermal images, to improve problems caused by reliance on auditors [8,12]. Previous studies have shown that thermal bridges can be detected from thermal images, based on thresholds such as the temperature difference and the shape of thermal bridges. However, threshold-based methods produce good results in certain cases, but there are difficulties in finding an optimized threshold applicable to all cases [14]. In the case of thermal images of building envelopes, the objects displaying abnormal temperature are different depending on the target and time of imaging. Furthermore, the thermal bridges in the thermal images always change in temperature and shape, depending on the circumstances. In such complicated situations, machine learning, rather than thresholds, can increase the detection accuracy in various cases [15,16].

This study proposes a method for detecting linear thermal bridges from thermal images based on machine learning. This article is structured as follows. Section 2 reviews studies on detecting thermal anomalies from thermal images. Section 3 describes the machine-learning-based thermal bridge detection method proposed in this study. Section 4 presents the results of applying this method to actual thermal images. Finally, Section 5 describes the conclusions of this study and future research.

2. Literature Review

Thermal imaging cameras were introduced to detect various thermal anomalies in building envelopes via non-destructive methods. In recent years, several studies have been performed to automatically detect thermal anomalies in building envelope by using image processing, in addition to thermal imaging cameras. Asdrubali et al. [8] detected thermal bridges from thermal images of specific areas, such as pillars and beam-pillar joints. This study proposed a method for determining the threshold to distinguish the temperature distribution in areas with normal and abnormal temperatures, to detect thermal bridges. Garrido et al. [12] conducted a study to automatically detect thermal bridges from thermal images by adding thresholds for geometric characteristics in addition to temperature differences caused by thermal bridges. As a result of applying the proposed method to thermal images, the precision, recall, and F-score were 55%, 68%, and 61%, respectively. As presented above, previous studies showed that areas classified as thermal bridges from thermal images can be recognized based on thresholds. However, there are difficulties in selecting a commonly used threshold when using threshold-based methods. There are also limitations in detecting and distinguishing thermal defects other than thermal bridges within thermal images.

In addition to architecture, a wide variety of studies have been performed to automatically detect problems, using thermal images in objects such as pipelines [16], turbojet engines [17], photovoltaic module [18,19], and belt conveyors [20]. In particular, some of these studies have proposed an intelligent method, such as machine learning, to increase the accuracy of detection. Tong et al. [16] proposed a method using machine learning, to detect leaks in pipelines. Thermal images of pipelines contain pipes and the background, so the Otsu–Grabcut image segmentation algorithm was applied to segment only the areas corresponding to pipes. Then, the Histogram of Oriented Gradient and Gray-Level Co-Occurrence Matrix were extracted from the segmented thermal images, as features for training. The extracted features were learned via a support vector machine. Through the proposed method, the normal pipeline, leaking pipeline, worn pipeline, and the normal valve were classified with an accuracy of 96.00%. Andoga et al. [17] proposed a method using machine learning to detect defects in a turbojet engine. The Kohonen neural network was applied to segment and recognize the defects. Through the proposed method, the defects were classified with an accuracy of 91.8%. As described above, various studies are being performed to apply image processing and machine learning techniques, to improve the accuracy of recognizing defects from thermal images. Since each application has different characteristics, each application requires a suitable method for segmenting images and extracting features to apply machine learning. However, such methods have not been proposed for thermal bridges in building structures.

3. Thermal Bridge Detection Method Using Neural Network

This study proposes a method for automatically detecting thermal bridges in building envelopes from thermal images. Figure 1 shows the method proposed in this study. First, the areas with thermal anomalies in the envelopes are clustered. Thermal anomaly area clustering involves three processes. The first process is clustering according to the temperature distribution, wherein areas with similar temperatures are considered as one cluster. Second, only clusters considered to be abnormal temperature distributions are segmented among the clusters with different temperature distributions. The results of classifying thermal anomaly areas are considered one cluster, even if the physical distance is distant, because clustering was performed by temperature distribution, as specified above. In the third process, clustering according to distance is performed, to recognize physically separated areas as different defects. The results of thermal anomaly area clustering allow us to obtain physically separated clusters with an abnormal temperature distribution. Features to detect thermal bridges are extracted from these individual clusters. Because thermal bridges have long vertical or horizontal shapes, this study selected linear distribution and complex boundaries as the features. After extracting the features, a pre-trained thermal bridge detection model is used to recognize areas corresponding to thermal bridges within the thermal image.

3.1. Thermal Anomaly Area Clustering

3.1.1. Temperature Clustering

Thermal anomalies in building envelopes, including thermal bridges, have a clear temperature difference from normal areas. These characteristics are used to detect the thermal anomalies of buildings through thermal images. Previous studies have confirmed that the temperature distribution of normal areas and abnormal areas with thermal anomalies follows a multi-modal Gaussian distribution [8,12]. The Gaussian Mixture Model (GMM) is a clustering method that configures multiple Gaussian components, to perform clustering [21]. Due to the nature of clustering based on Gaussian components, this study applied GMM for clustering between similar temperature distributions.

The temperature clustering based on GMM, proposed in this study, is performed as follows. First, clustering is performed by applying GMM with the temperature of each pixel constituting the thermal image as input. As shown in Figure 2a, the results produce mixture components with K Gaussian distributions. Figure 2a shows the temperature distribution histogram for four mixture components. Figure 2b shows the location of each distribution in the thermal image, and the colors between Figure 2a,b correspond to each other. As shown in the figure, temperatures belonging to the same mixture component are regarded as clusters with the same temperature characteristics. The Bayesian Information Criterion (BIC) is applied to determine K, which represents the number of mixture components. BIC is the most widely used method to determine the optimized K of GMM [22]. Since the K-value to minimize BIC is the optimized K-value in GMM fitting, this study set the range for finding the optimal K from 2 to 8 and selected the K-value with the lowest BIC value.

3.1.2. Thermal Anomaly Area Segmentation

Thermal images of building envelopes contain both areas with normal temperature distributions and areas with abnormal temperature distributions. In general, areas with normal temperature distributions occupy a large proportion within the thermal image, and areas with abnormal temperature distributions cover a relatively small area. Based on this assumption, this study segmented areas with abnormal temperature distributions based on the temperature of each cluster.

As mentioned above, GMM-based clustering allows us to obtain the temperature of the pixels included in each cluster. The average temperature in each cluster is calculated by using the temperature of the pixels in each cluster. After calculating the average temperature of each cluster, the number of pixels in each cluster is counted. The cluster with the most pixels is regarded as the area corresponding to the normal outer wall. As shown in Figure 3a, the clusters are distinguished with an abnormal temperature distribution by reflecting the season based on the average temperature µ2 of the cluster with the most pixels. For example, during winter, the normal outer wall shows a relatively lower temperature than areas with thermal anomalies, such as thermal bridges. Therefore, in the case of thermal images measured in winter, clusters with an average temperature higher than the average temperature of the cluster with the most pixels are classified as abnormal areas. In terms of thermal images measured in summer, clusters with an average temperature lower than the average temperature of the cluster with the most pixels are classified as abnormal areas. Since the example in Figure 3 was measured in winter, clusters with average temperatures (µ3 and µ4) higher than µ2, the average temperature of the cluster with the most pixels, are classified as abnormal clusters. Figure 3b shows an example of abnormal temperature areas segmentation.

3.1.3. Thermal Anomaly Clustering

This study applied Density-Based Spatial Clustering of Applications with Noise (DBSCAN) [23], to segment physically separated clusters. DBSCAN is a density-based clustering method that has been successfully applied in many real-world applications [24]. In particular, it is designed to find clusters in noisy data [25].

The segmentation process of abnormal area pixels by applying DBSCAN is as follows. First, the pixels in the abnormal areas are converted into 2D coordinate values. The pixels converted into 2D coordinate values have x and y coordinates, respectively. After selecting a random starting point among the points converted into 2D coordinates, the neighboring points located within a certain distance (ε) are identified from this starting point. If the number of points within ε is equal to or greater than the minimum number of points (MinPts), the corresponding point and neighboring point are considered as one cluster. Then, the process is repeated for all the neighbors. If the number of neighboring points is less than MinPts, the corresponding point is considered as noise. After repeating this for all the neighbors, the procedure above is repeated for all the remaining points. Figure 4 below shows the clustering process of pixels segmented as abnormal areas by applying DBSCAN.

3.2. Feature Extraction

3.2.1. Data Distribution Linearity

Thermal bridges in building envelopes are caused by structural insulation loss and mainly occur at the joints between wall–slab or wall–wall structures. Therefore, thermal images of the outer structure of buildings show long thermal bridges in a specific direction, such as the cross-sectional shape of slabs and walls. This study selected the ratio of the main axis in which data are distributed and the data perpendicular to the main axis as the first feature to reflect the linear shape of thermal bridges. Even if the shape of the thermal bridge is not vertical or horizontal in the thermal image, the feature proposed in this study can be applied robustly, because it calculates the main axis according to the distribution of data.

To extract the features of the main axis and data distribution of the clusters segmented above, a covariance matrix is calculated by using the 2D coordinates of the data included in each cluster. The covariance matrix is calculated as follows:

$$\mathrm{Covariance}\mathrm{matrix}= \frac{1}{N}{\displaystyle \sum }_{i=1}^N\left(X_i-\overline{X}\right){\left(X_i-\overline{X}\right)}^T$$

(1)

where N is the number of points in each cluster, $X_i$ is the coordinates of the i-th point, and $\overline{X}$ is the average of the coordinates of all the points in the cluster.

The eigenvector and eigenvalue of each cluster can be obtained through decomposition of the covariance matrix. This eigenvector is the principal component vector, which shows the direction in which the variance is large in the distribution of the data, and the corresponding eigenvalue indicates the size of the variance in the that direction. This study extracted the first feature value $F_1$ by calculating the ratio of the eigenvalue corresponding to the first eigenvector and the eigenvalue corresponding to the second eigenvector. The first feature is calculated as follows:

$$F_1= \frac{{\lambda }_2}{{\lambda }_1}$$

(2)

where ${\lambda }_1$ is the eigenvalue corresponding to the first eigenvector, and ${\lambda }_2$ is the eigenvalue corresponding to the second eigenvector.

3.2.2. Complexity of Data Boundaries

The thermal bridges in thermal images show simple trends similar to straight lines without complex boundaries in addition to linear shapes. This study proposes the ratio of actual data relative to the area surrounding individual clusters to extract features with uncomplicated boundary surfaces. The features of the trend with complex data boundaries proposed in this study are calculated as follows. First, the convex hull algorithm is applied to calculate the area surrounding the individual clusters. The convex hull algorithm is defined as a convex polygon that includes all the points in a binarized image [26]. The convex hull algorithm is widely used to extract specific shapes within an image. Figure 5 below shows an example of applying convex hull in a 2D image. The line surrounding the cluster represents the convex hull of that cluster, and the black shading represents the points constituting the cluster.

A convex polygon that contains the points constituting each cluster can be obtained by calculating the convex hull. Based on the convex polygons, the area occupied by the convex polygons constituting each cluster is calculated. Among the areas occupied by the convex polygons, the proportion of the actual points depends on the distribution of the points in the cluster. As shown in Figure 5a, in the case of thermal bridges, the actual points account for a large proportion of the area of the convex polygon. However, in the case of clusters that are not thermal bridges, the actual points account for only a small proportion of this area, as shown in Figure 5b. Since each pixel has an area of 1 by 1, this study calculated feature $F_2$, which determines the complexity of data boundaries, as the number of pixels relative to the area enclosed by a convex hull. The second feature is calculated as follows:

$$F_2= \frac{N_p}{A_c}$$

(3)

where $A_c$ is the area of the convex polygon, and $N_p$ is the number of points in the cluster.

3.3. Thermal Bridge Modeling

The purpose of this study was to detect thermal bridges from thermal images, which have morphologically distinct features that are slightly different from case to case, making them difficult to detect by only using thresholds. An artificial neural network was applied to achieve this purpose. Because of their ability to solve non-linear problems, artificial neural networks (ANNs) are used in applications for predicting the relationship between complex inputs and outputs including subjective judgment and experience [27].

The ANN is a method developed based on the behavior and structure of the human brain [28]. The ANN is a mathematical model depicting biological neurons. A typical ANN model consists of an input layer, an output layer, and a hidden layer [29]. Neurons are arranged in each layer and fully connected with weights that affect results [30]. Each weight is defined in an optimization way to increase the accuracy in predicting target values. For the non-linear feature of the transfer function between neurons, the ANN is also good at solving the non-linear problems.

In the case of thermal images including thermal bridges, images of actual thermal bridges must be obtained because there is no large-scale public database. Therefore, this study constructed a structure for learning small datasets because of the lack of training datasets for thermal-bridge modeling. The training process was performed by extracting features from images of previously collected thermal bridges and other thermal-anomaly areas. This study applied a backpropagation neural network, which is appropriate for processing small amounts of data to train the ANN [31]. Among the training parameters, the number of hidden layers has a significant influence on prediction results [32]. This study set the number of hidden layers to one by considering the number of input and output layers. The sigmoid function was applied as the activate function of the hidden layer. Figure 6 shows the structure of the ANN in this study. The datasets for training were randomly divided into training at 70%, validation at 15%, and testing at 15%, which is a good ratio for small datasets [33].

4. Experimental Results

4.1. Collection of Thermal Bridge Data

Thermal images of old houses were acquired to develop a model for classifying thermal bridges. Thermal images were taken when environmental conditions suggested by ISO 6781 [34] were satisfied. The images were taken at dawn before sunrise because thermal images can be influenced by solar radiation. In addition, the images were taken in winter because a high temperature difference between the indoor and outdoor may distinguish thermal bridges from other normal thermal images. Actually, the average outdoor and indoor temperature were −1.02 and 12.78 °C, respectively. This satisfied the indoor and outdoor temperature difference of 10 °C or more, following ISO 6781. Finally, all the measurements were achieved with a wind speed of less than 5 m/s. This study chose the conventional walk-through thermal imaging method to take the images. In this method, a person walks around the target to check the building envelope with thermal images and takes images of thermal anomalies. In this study, the thermographer walked around the target building and took images of all the thermal anomalies in the building envelope, even if they were not thermal bridges. In the conventional method, the audit records while moving around the building and interprets the pattern based on the apparent temperature [13]. Therefore, in this study, the apparent temperature was measured by using default parameters for emissivity and reflected temperature. This study used a Testo 882 thermal imaging camera, the specifications of which are shown in

Table 1. Specification of infrared thermal imaging camera used to collect thermal bridge data.

Title	Specification
Measurement range	−20 °C~+550 °C
Accuracy	±2 °C or ±2%
Thermal sensitivity	0.005 °C at 30 °C
Wavelength range	8~14 μm
IR resolution	320 × 240 pixels

.

After taking the thermal images, the thermal bridges and the other anomalies were classified, cropped, and used as data for training. This study acquired a total of 134 thermal images for thermal bridge modeling. These thermal images contain thermal anomaly areas. A total of 151 thermal bridge cases and 223 other cases were cropped. All of these data were used for thermal bridge modeling. In addition to the thermal images used for thermal bridge modeling, eight more thermal images were acquired and used to validate the methodology proposed in this study.

4.2. Evaluation Metrics

To validate the methodology proposed in this study, the results were compared with the ground truth. To create the ground truth, the author checked each thermal image and designated the areas corresponding to thermal bridges. The comparison between the thermal bridge detection results and the ground truth was measured through precision, recall, and F-score. These three metrics have been used to validate methodologies for detecting thermal anomalies in thermal images, including thermal bridges [12] and moisture [35]. All three metrics are calculated as true positive, false positive, and false negative. True positive refers to the number of pixels predicted as thermal bridges that are actual thermal bridges. False positive refers to the number of pixels predicted as thermal bridges that are not actual thermal bridges. False negative refers to the number of pixels predicted not to be thermal bridges that are actual thermal bridges. Precision, recall, and F-score are calculated as follows:

$$\mathrm{Precision}= \frac{TP}{TP+FP}$$

(4)

$$\mathrm{Recall}= \frac{TP}{TP+FN}$$

(5)

$$\mathrm{F}-\mathrm{score}= 2\times \frac{Precission\times Recall}{Precision+Recall}$$

(6)

where TP is true positive, FP is false positive, and FN is false negative.

4.3. Results

Figure 7 shows the thermal images, segmentation results, and thermal bridge detection results used for validation. The segmentation results are the results of performing GMM-based clustering, thermal anomaly area detection, and DBSCAN-based clustering. In Figure 7, each cluster is displayed in different colors, and the pixels of the same color are in the same cluster. When the thermal images only displayed a wall, as in Cases 3, 6, and 8, the areas corresponding to thermal bridges were segmented well. However, as shown in Cases 1, 2, 4, 5, and 7, if a thermal image contains areas that appear to be thermal defects due to low thermal performance, such as windows and walls, the image contains clusters that are both thermal bridges and not thermal bridges.

When there are clusters that are thermal bridges and not thermal bridges, clusters corresponding to thermal bridges can be classified by performing machine-learning-based classification after feature extraction. Figure 7c shows the thermal bridge detection results for each case. In the detection results, the areas highlighted in green represent the thermal bridge areas detected by the proposed method. As shown in the results, when there is only one thermal bridge, as in Case 1, Case 2, Case 3, Case7, and Case 8, only one cluster was detected as a thermal bridge in the detection results. In Case 6, where there are two thermal bridges, two clusters were detected as thermal bridges. However, in Case 4 and Case 5, the areas where heat was generated because of the window frame and heating pipe were classified as thermal bridges, but they actually were not thermal bridges. This was because these areas had similar shapes as thermal bridges. The results below show that the thermal bridge detection method proposed in this study can automatically detect multiple thermal bridges, even when thermal images contain patterns that appear to be thermal defects other than thermal bridges.

Table 2. Precision, recall, and F-score of the proposed thermal bridge detection method.

Case Number	Precision (%)	Recall (%)	F-Score (%)
1	94.52	71.46	81.39
2	89.62	94.44	91.97
3	85.33	91.69	88.39
4	78.12	86.64	82.16
5	75.97	99.08	86.00
6	94.93	89.54	92.12
7	96.60	73.68	83.60
8	99.25	91.81	95.38
Average	89.29	87.29	87.63

shows the comparison with the ground truth after applying the proposed method to eight thermal images. The results vary from case to case, but the maximum and minimum precisions were 99.25% and 75.97%, respectively. In terms of cases with low precision, several pixels were recognized as thermal bridges that were not actual thermal bridges, as shown in Cases 4 and 5 in Table 2. This is because the upper frame of the window and the outer wall area adjacent to a hot water pipe, which displayed similar shapes as thermal bridges, were recognized as thermal bridges. The maximum and minimum recalls were 99.08% and 71.46%, respectively. In Case 1, which has a low recall, there are many areas of pixels that are not recognized as thermal bridges, even though they are thermal bridges. This is because they were segmented into different clusters due to temperature differences during the segmentation process. In terms of F-Score, the minimum score was 81.39%, and all the other cases were more than 80%. The average precision, recall, and F-score of the eight thermal images used for validation were 89.29%, 87.29%, and 87.63%, respectively. Although the validation targets are different, the detection results showed improvement, compared to the results of previous studies, as follows: precision, 55%; recall, 68%; and F-score, 61%.

5. Conclusions

This study proposed a new method for detecting linear thermal bridges from thermal images, using image processing and ANN. After performing GMM-based clustering, using temperature, the average temperature of each cluster was calculated, to compare with normal walls. Based on winter measurements, areas with higher temperatures than normal walls were classified as thermal anomaly areas, and DBSCAN was applied to divide the thermal anomaly areas into individual defects, to perform density-based clustering. Subsequently, feature values reflecting morphological characteristics of thermal bridges were extracted from each cluster. Based on the extracted feature values, only thermal bridges were classified by using a trained model, using ANN. A field experiment was performed to validate the thermal bridge detection method proposed in this study. Thermal images were taken of buildings with actual thermal bridges, and the precision, recall, and F-score were calculated. As a result, the average precision, recall, and F-score were 89.29%, 87.29%, and 87.63%, respectively.

The contributions of this study are as follows. First, it proposed a feature extraction method that reflects the morphological characteristics of thermal bridges. The proposed features can be used in other fields that require feature extraction of objects with linear or flat shapes, in addition to thermal bridges. Second, this study proposed a thermal bridge detection process using machine learning. Finally, from a practical aspect, the method proposed in this study secures objectivity by automatically detecting thermal bridges, which relied on subjective judgment in conventional audits. This study proposed a method for detecting only thermal bridges among various thermal defects in building envelopes. Further research will be conducted to propose a method to detect defects other than thermal bridges.