Defect Detection by Analyzing Thermal Infrared Images Covered with Shadows with a Hybrid Approach Driven by Local and Global Intensity Fitting Energy ^†

Abstract

Defect detection using thermal infrared images is used in nondestructive evaluation and testing because it is easy to use. Thermal infrared images recorded the surface temperatures of the target with a thermal infrared camera. Image segmentation is a technique to group those pixels with similar surface temperatures to form thermal patterns. Defects can be identified by the segmented patterns having different surface temperatures in their neighborhoods. In this study, a hybrid approach combining fitting energy is proposed for describing the contamination illustrated in the recorded surface temperatures and regional constants averaging the surface temperatures of the segmented regions. The numerical implementation is completed by applying the level set functions on an iteration scheme. The initial level sets evolve till a convergence can be reached. The processed results demonstrate that the hybrid approach can be used for defect detection.

1. Introduction

Defect detection is an essential issue in nondestructive testing (NTD). Thermal infrared images are widely employed by identifying the recorded surface temperatures presented in the given thermal infrared images. Defects are usually identified by finding the differences in surface temperatures. However, the recorded surface temperatures presented in the thermal infrared images can be contaminated by sunlight refraction and environmental deficiencies like shadows. Those contaminations make the pixels of the thermal infrared image contain not only the surface temperature but also extra information related to those contaminations.

Removing the intensity inhomogeneity presented in the given thermal infrared images is an important issue in analyzing thermal infrared images. Huang et al. modeled the shadow effects in a multiplicative way, and the shadow effects can be approximated by implementing level sets and iteration schemes [1]. Traditionally, Li et al. introduced the local fitting energy to model the intensity inhomogeneity in their segmentation algorithm [2]. Zhang et al. employed regional standards and regional constants to limit the segmentation regions [3]. Zhang et al. proposed a hybrid approach combining the fitting energy and regional constants to segment the given images [4].

Image segmentation grouping those pixels with similar surface temperatures is employed to identify the defects by finding the differences in the surface temperature. The authors used their algorithms to approximate the intensity inhomogeneity and simultaneously segment the given images. Huang et al. successfully used Zhang’s regional standard deviations and regional constants to locate those potential defects [1]. They employed a hybrid approach combining the local fitting energy and regional parameters (like standard deviations and regional constants) to approximate intensity inhomogeneity. The processed results demonstrated that the proposed hybrid approach can be used for defect detection. The paper is organized as follows. Section 2 introduces the hybrid system, Section 3 presents a series of thermal infrared images of the side wall of the Administration Building, Chaoyang University of Technology, Taichung, Taiwan, as the test target to verify the robustness of the proposed approach, and Section 4 provides the discussions and conclusions.

2. Hybrid System

2.1. Principal Component Analysis

Principal component analysis (PCA) is widely used to analyze a series of images by projecting the original data onto a low-dimension space. The projected data can inherit the major properties from the original data. In doing so, the template image extracted from the projected data can be analyzed instead of analyzing each given image. Suppose a given data matrix M can be decomposed as a low-dimension space L and a sparse space S. Then, M can be given as follows.

$$\mathrm{M}=\mathrm{L}+\mathrm{S}$$

(1)

Equation (1) needs to satisfy the given condition as follows.

$$\min \Vert \mathrm{M}-\mathrm{L}\Vert \mathrm{st}.\mathrm{rank}\left(\mathrm{L}\right)\le \mathrm{k}$$

(2)

where $\mathrm{k}<\mathrm{rank}\left(\mathrm{M}\right).$ The singular value decomposition (SVD) is applied to find the optimal approximation.

2.2. Hybrid Systems

The proposed hybrid system contains the local fitting energy and regional parameters (including regional standard deviations and regional constants). The local fitting energy assumes that the intensity inhomogeneity illustrated in the given image can be approximated by the local fitting energy [2]. Hence, the local fitting energy can be given as follows [2].

$$E_1\left(\mathsf{\Phi },F,C\right)={\displaystyle \sum }_{i=1}^4{\displaystyle \iint }{K}_{\sigma }\left(x-y\right){\left|I\left(y\right)-f_i\left(x\right)\right|}^2{M}_i\left(y\right)dydx$$

(3)

where $K_{\sigma }\left(x-y\right)$ is Gaussian filter with the parameter $\mathsf{\sigma }$, I is the given image, $f_i$ is the local fitting energy and ${\mathrm{M}}_{\mathrm{i}}$ are the combinations of two-level set functions. ${\mathrm{M}}_{\mathrm{i}}$ is presented as follows.

$$\begin{gathered} M_1\left({\varphi }_1,{\varphi }_2\right)=H\left({\varphi }_1\right)H\left({\varphi }_2\right) \\ {M}_2\left({\varphi }_1,{\varphi }_2\right)=H\left({\varphi }_1\right)\left(1-H\left({\varphi }_2\right)\right) \\ {M}_3\left({\varphi }_1,{\varphi }_2\right)=\left(1-H\left({\varphi }_1\right)\right)H\left({\varphi }_2\right) \\ {M}_3\left({\varphi }_1,{\varphi }_2\right)=\left(1-H\left({\varphi }_1\right)\right)\left(1-H\left({\varphi }_2\right)\right) \end{gathered}$$

(4)

where ${\varphi }_1,{\varphi }_2$ are level set functions, and H indicates the Heaviside function, shown as follows [5].

$$H_{\epsilon }\left(x\right)=\frac{1}{2}\left[1+\frac{2}{\pi }ta{n}^{-1}\left(\frac{x}{\epsilon }\right)\right]$$

(5)

where ε > 0.

The image model containing the intensity inhomogeneity is incorporated in a multiplicative way and regional constants. The image model is illustrated as follows [3].

$$\mathrm{I}={\displaystyle \sum }_{i=1}^4{\mathrm{BC}}_{\mathrm{i}}$$

(6)

where B is the intensity inhomogeneity, and C_i is the regional constant. The regional constants are the average values of the segmented regions. Zhang et al. introduced the standard deviations of the segmented regions into the segmentation algorithm, and the algorithm can be presented as follows [3].

$$E_2\left(\sigma ,c,B\right)=\int \left({\displaystyle \sum }_{\mathrm{i}=1}^4\int \mathrm{K}\left(\mathrm{y},\mathrm{x}\right)\left(\log {\mathsf{\sigma }}_{\mathrm{i}}+\frac{{\left(\mathrm{I}-\mathrm{B}{C}_{\mathrm{i}}\right)}^2}{2{\mathsf{\sigma }}_{\mathrm{i}}^2}\right){\mathrm{M}}_{\mathrm{i}}\left(\mathsf{\Phi }\right)\mathrm{dx}\right)\mathrm{dy}$$

(7)

where ${\sigma }_i$ are the regional standard deviations of the segmented regions. The hybrid system is the linear combination of Equations (3) and (7) and is given as follows.

$$\begin{gathered} E=\omega E_1+\left(1-\omega \right)E_2 \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\nu {\displaystyle \int }\left|\nabla H\left({\varphi }_1\right)\right|dx \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \hspace{1em}\hspace{1em}\hspace{1em}+\nu {\displaystyle \int }\left|\nabla H\left({\varphi }_2\right)\right|dx+\mu \int \frac{1}{2}{\left(\left|\nabla {\varphi }_1\right|-1\right)}^{2}dx \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\mu \int \frac{1}{2}{\left(\left|\nabla {\varphi }_2\right|-1\right)}^{2}dx \end{gathered}$$

(8)

where $\omega , \nu , \mathrm{and} \mu$ are positive constants.

Then, the local fitting energy can be obtained and given as follows.

$$f_i\left(x\right)=\frac{K_{\sigma }\left(x-y\right)⭙\left(I{M}_i\left(\mathsf{\Phi }\right)\right)}{K_{\sigma }\left(x-y\right)⭙M_i\left(\mathsf{\Phi }\right)}$$

(9)

where ⭙ is the convolution operator. Similarly, the regional constants, regional standard deviations, and the intensity inhomogeneity illustrated in the image can be given as follows.

$$C_i=\frac{\int K_{\sigma }\left(y,x\right)⭙\left(IB{M}_i\left(\mathsf{\Phi }\right)\right)dxdy}{\int K_{\sigma }\left(y,x\right)⭙\left(B^2{M}_i\left(\mathsf{\Phi }\right)\right)dxdy}$$

(10)

$${\sigma }_i^2=\frac{\int K\left(y,x\right)⭙\left({\left(I-B{C}_i\right)}^2{M}_i\left(\mathsf{\Phi }\right)\right)dxdy}{\int K\left(y,x\right)⭙M_i\left(\mathsf{\Phi }\right)dxdy}$$

(11)

$$B=\frac{{\sum }_{i=1}^4\int K_{\sigma }\left(y,x\right)⭙\left(I{M}_i\left(\mathsf{\Phi }\right)\frac{C_i}{{\sigma }_i^2}\right)dxdy}{{\sum }_{i=1}^4\int K\left(y,x\right)⭙\left(M_i\left(\mathsf{\Phi }\right)\frac{C_i^2}{{\sigma }_i^2}\right)dxdy}$$

(12)

The iteration scheme is applied such that the level set functions can evolve till the convergence is reached. Firstly, several parameters are defined as follows.

$$\begin{gathered} e_1=\iint K_{\sigma }\left(x-y\right){\left|I\left(y\right)-f_1\left(x\right)\right|}^{2}dydx \\ {e}_2=\iint K_{\sigma }\left(x-y\right){\left|I\left(y\right)-f_2\left(x\right)\right|}^{2}dydx \\ {e}_3=\iint K_{\sigma }\left(x-y\right){\left|I\left(y\right)-f_3\left(x\right)\right|}^{2}dydx \\ {e}_4=\iint K_{\sigma }\left(x-y\right){\left|I\left(y\right)-f_4\left(x\right)\right|}^{2}dydx \\ {F}_1=\int K\left(y,x\right)\left(\log {\sigma }_1+\frac{{\left(I-B\left(x\right)C_1\right)}^2}{2{\sigma }_1^2}\right)dxdy \\ {F}_2=\int K\left(y,x\right)\left(\log {\sigma }_2+\frac{{\left(I-B\left(x\right)C_2\right)}^2}{2{\sigma }_2^2}\right)dxdy \\ {F}_3=\int K\left(y,x\right)\left(\log {\sigma }_3+\frac{{\left(I-B\left(x\right)C_3\right)}^2}{2{\sigma }_3^2}\right)dxdy \\ {F}_4=\int K\left(y,x\right)\left(\log {\sigma }_4+\frac{{\left(I-B\left(x\right)C_4\right)}^2}{2{\sigma }_4^2}\right)dxdy \end{gathered}$$

(13)

The level set functions are rewritten for the time parameter and can be presented as follows.

$$\begin{gathered} \frac{\partial {\varphi }_1}{\partial t}=\omega \delta \left({\varphi }_1\right)H\left({\varphi }_2\right)\left(e_3-e_1\right)+\omega \delta \left({\varphi }_1\right)\left(1-H\left({\varphi }_2\right)\right)\left(e_4-e_2\right)+\left(1-\omega \right)\delta \left({\varphi }_1\right)H\left({\varphi }_2\right)\left(F_3-F_1\right) \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\left(1-\omega \right)\delta \left({\varphi }_1\right)\left(1-H\left({\varphi }_2\right)\right)\left(F_4-F_2\right)+\nu \delta \left({\varphi }_1\right)div\left(\frac{\nabla {\varphi }_1}{\left|\nabla {\varphi }_1\right|}\right)+\mu \left({\nabla }^2{\varphi }_1-div\left(\frac{\nabla {\varphi }_1}{\left|\nabla {\varphi }_1\right|}\right)\right) \end{gathered}$$

(14)

$$\begin{gathered} \frac{\partial {\varphi }_2}{\partial t}=\omega \delta \left({\varphi }_2\right)H\left({\varphi }_1\right)\left(e_2-e_1\right)+\omega \delta \left({\varphi }_2\right)\left(1-H\left({\varphi }_1\right)\right)\left(e_4-e_3\right)+\left(1-\omega \right)\delta \left({\varphi }_2\right)H\left({\varphi }_1\right)\left(F_2-F_1\right) \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\left(1-\omega \right)\delta \left({\varphi }_2\right)\left(1-H\left({\varphi }_1\right)\right)\left(F_4-F_3\right)+\nu \delta \left({\varphi }_2\right)div\left(\frac{\nabla {\varphi }_2}{\left|\nabla {\varphi }_2\right|}\right)+\mu \left({\nabla }^2{\varphi }_2-div\left(\frac{\nabla {\varphi }_2}{\left|\nabla {\varphi }_2\right|}\right)\right) \end{gathered}$$

(15)

In doing so, the intensity inhomogeneity can be approximated. Furthermore, the image can be calibrated by removing the intensity inhomogeneity.

3. Materials for Evaluation

A series of thermal infrared images were taken on 30 January 2019. Those thermal infrared images were recorded by NEC InfReC R500Pro, and the image sizes of each recorded image are 480 by 640 pixels. The accuracy of the recorded surface temperature reached 0.01 °C. The test target is the side wall of the Administration Building, Chaoyang University of Technology, Taichung, Taiwan. There were 80 frames recorded by NEC InfReC R500Pro, and the highest surface temperature and the average surface temperature in each recorded thermal infrared image are presented in Figure 1. Figure 2 shows the images recorded by NEC InfReC R500Pro and a digital camera installed on the thermal infrared camera.

Then, PCA was applied to the recorded thermal infrared images, and the first image was extracted from the low-dimension space. The extracted image is illustrated in Figure 3. It is obvious that the information on the surface temperatures was lost because the projected data onto a low-dimension space could not keep the temperature information. However, the thermal patterns remained.

The proposed hybrid system with settings $\omega =0.1$, $\mu =0.00001\times 256\times 256$, $\nu =1$, and $∆t=0.1$ was used on the results by employing PCA. The approximated intensity inhomogeneity is presented in Figure 4. The segmented results are shown in Figure 5. The calibrated image with removing intensity inhomogeneity is illustrated in Figure 6. The convergence is given in Figure 7.

4. Discussions and Conclusions

The proposed hybrid system can segment the given thermal infrared images such that the differences in the recorded surface temperatures can be identified. From Figure 5, it is evident that those segmented regions colored in yellow can be potential defects. Different NDT techniques can be employed to verify the results.

The proposed hybrid system employs the Gaussian function to assume that the intensity inhomogeneity is illustrated at location x and in its neighborhoods. The Gaussian function shows that for those neighborhoods, their influences decrease while their locations are far away from location x. With the specified $\sigma$, the Gaussian function with different window sizes is applied in the hybrid system. The window sizes are 5 × 5, 15 × 15, 25 × 25, and 35 × 35. The performance is presented in

Table 1. Performances of Employing Different Window sizes.

Window Sizes	Processing Time (s)
5 × 5	256.37
15 × 15	334.43
25 × 25	457.79
35 × 35	610.10

. The processing times were calculated by taking the averages after running the same program ten times. The processing time was increased with the window sizes. The estimated intensity inhomogeneity is presented in Figure 8a–d.

The intensity inhomogeneity does exist in the thermal infrared images, and its ranges are in the [0, 1.4]. The small window sizes, like 5 × 5, seem to have bigger ranges than those larger window sizes because the small window sizes give the influences from the neighborhoods such that the ranges are increased. Shadows illustrated in the thermal infrared images have intensity inhomogeneity, and the effects can be approximated. Furthermore, the intensity inhomogeneity can be removed. The estimated intensity inhomogeneity enhances the original image, while the intensity inhomogeneity is less than 1.0. Otherwise, the image is smoothed while the intensity inhomogeneity is larger than 1.0. As for that intensity inhomogeneity equal to 1.0, nothing can be done on the images.

Eventually, the conclusions are given as follows.

(1)

Image segmentation can be employed to find potential defects by segmenting the given thermal infrared images.

(2)

PCA can project the given data onto a low-dimension space such that the properties of the given data can be inherited from the images extracted from the low-dimension space.

(3)

Intensity inhomogeneity does exist, and it needs to be estimated such that the thermal infrared images can be calibrated.

The proposed hybrid system seems to work well in analyzing thermal infrared images. Different methods to remove intensity inhomogeneity will be compared in the future.