*3.2. Photometric Stereo System*

PTZ is one type of module used for NPP visual inspection. In our configuration, it is equipped with four 30 W LED lights placed around a CMOS camera with 1920 × 1080 picture elements, attached with an F1.6~F3.0 lens, as shown in Figure 1. The LEDs exhibit approximately the same performance in terms of the radiant flux value and emitting angle. In the PTZ setup, all LED lamps are fixed to make their optical axes parallel to the viewing angle of the camera. Each LED can be dimmed in increments from 0% to 100%. In addition, two laser generators display reference points for length measurement; the distance between them is calibrated prior to the experiment. Practical inspections must be conducted at a viewing angle as perpendicular to the target surface as possible. As shown in Figure 2, *Xc*, *Yc*, and *Zc* are the three axes of the camera coordinate system, which also represent the global coordinate system in our method.

**Figure 1.** Pan/tilt/zoom (PTZ) image capture setup and light-emitting diode (LED) light layout dimensions (mm).

**Figure 2.** Diagram of the photometric stereo system.

The first stage of the shape reconstruction algorithm—i.e., the calibration of camera parameters—begins by calibrating the camera intrinsic parameters using Toolbox in Matlab. Then, images are captured using the camera with the laser points turned on. Given these images, the following steps are taken:


The PS formulation for our configuration is briefly introduced in Section 2, and details of the algorithm are provided in Sections 3.3 and 3.4.

#### *3.3. Estimation of Light Direction and Intensity*

Previous studies typically assume that the light sources are infinitely far from the surface, and generally adopt the parallel ray model. However, our system uses LEDs, which are very close to the target surface; thus, the lighting direction and intensity differ among various image areas. Moreover, the camera and lights are not fixed during an actual inspection. As previous light calibration methods are neither practical nor accessible for our application, it is necessary to design a fully automatic calibration method to estimate the light direction and intensity at each point.

In our configuration, the viewing angle is as perpendicular to the target surface as possible. In addition, the target surface is approximated as a planar surface that is assumed to be parallel to the image plane. Furthermore, the LED chip center and camera lens lie on a single plane, which is also parallel to the image plane. Accordingly, there are three parallel planes in the configuration, as shown in Figure 2.

The lighting direction for each point is decided by the position of the light *Lp* and the point *P* on the target surface. Supposing that the camera plane is the horizontal plane in the coordinate system, *Lp* can be determined by the PTZ structure. To determine the coordinate of point, the mapping relationship between image pixels and surface points must be determined, as well as the distance between the camera plane and the target plane. Thus, a two-stage process is designed:


Figure 2 shows that the viewing angle is aligned with the negative *z*-axis of the coordinate system, which simplifies the geometry calculation. The laser generators are fixed on PTZ, and their relative positions are known. Thus, *P*1(*x*1, *y*1, *z*) and *P*2(*x*2, *y*2, *z*), the intersections of the laser optical axis and the target plane, are also fixed. Their corresponding image pixels are *I*<sup>1</sup> *x* <sup>1</sup>, *y* <sup>1</sup>, *z* and *I*<sup>2</sup> *x* <sup>2</sup>, *y* <sup>2</sup>, *z* , respectively, as shown in Figure 2.

The assumption of orthographic projection has typically been used in the conventional PS, although the perspective projection has been demonstrated to be more realistic [24]. However, when the change in scene depth is small relative to the distance from the scene to the camera, an orthographic projection can be used instead of a perspective projection [25]. In our method, the viewing angle is kept as perpendicular to the target surface as possible, and the distance between the target surface and the camera is considerably greater than the depth of the defects. Therefore, it is reasonable to apply an orthographic projection model without resulting in a large deviation.

The image magnification can be expressed as follows:

$$m = \frac{\|\|P\_1 - P\_2\|\|}{\|\|I\_1 - I\_2\|\|} = \frac{f}{z} \tag{3}$$

where *m* is the imaging magnification, *f* is the camera focal length, denotes the length of a vector, and *z* is the distance between the target surface and the lens aperture, which is approximately equal to the distance between the surface plane and the camera plane. We denote *z*<sup>0</sup> as the calibrated distance of reference *I*<sup>1</sup> *x*1, *y*1, *z* and *I*<sup>2</sup> *x*2, *y*2, *z* ; *z* can then be calculated using

$$z = \frac{\left[\left(\mathbf{x}\_2 - \mathbf{x}\_1\right)^2 + \left(\mathbf{y}\_2 - \mathbf{y}\_1\right)^2\right]^{1/2}}{\left[\left(\mathbf{x}\_2' - \mathbf{x}\_1'\right)^2 + \left(\mathbf{y}\_2' - \mathbf{y}\_1'\right)^2\right]^{1/2}} z\_0. \tag{4}$$

Laser points with different distances between the camera plane and the target plane are shown in Figure 3. Thus, the next step is to define how a point on the target surface is related to a pixel on the image plane.

**Figure 3.** Laser points with different distances between the camera plane and the target plane: (**a**) 200 mm and (**b**) 350 mm.

As illustrated in Figure 2, there is a clear relationship between image pixels and their corresponding points. For any image pixel *I x i* , *y i* , *z* , the position of its corresponding point in the target surface can be defined as *P kx i* , *ky i* , *z* , where *k* = 1/*m*. Therefore, the coordinate of *P* is decided by *k*, which can be determined by the laser points and *z*. The position of the LED,*LP xLED*, *yLED*, 0 , can also be estimated from prior knowledge. Therefore, the light direction *<sup>l</sup>*, or <sup>→</sup> *LP*, can be expressed as *kx <sup>i</sup>* − *xLED*, *ky <sup>i</sup>* − *yLED*, *z* .

For LED lights, the irradiance (*E*) on the target surface can be expressed as follows:

$$E = \frac{I\_{LED}(\theta)\cos(\theta)}{r^2} \tag{5}$$

where θ is the emitting angle of the LED, *ILED* denotes the radiant intensity of the LED, and *r* is the distance from the light source to the target point [17]. By transforming the parameters, *E* can also be calculated from Equation (5). Then, *l* and *E* can be determined for various image pixels *I x i* , *y i* .

The goal of this step is to determine the lighting direction and intensity for each image pixel, which will improve the accuracy of the surface normal calculation, as well as the final 3D data quality.

#### *3.4. Three-Dimensional Reconstruction using Photometric Stereo Technique*

The PS procedure, assuming a Lambertian reflectance model, is applied for 3D shape reconstruction. The normal *n* is determined by using at least three images with various lighting conditions. In this study, they are calculated using the illustrated lighting direction and intensity estimation methods. Then, the image intensity follows inverse squared law, as

$$I = \rho \frac{(n \times l)}{r^2} = \rho \frac{n \times \left(P - L\_p\right)}{\|\|P - L\_p\|\|^{3/2}}\tag{6}$$

where *r* is the distance between the light source and the surface point [17].

The normal, *n*, can be calculated using at least three equations; once it is solved, the surface shape can be reconstructed via the height estimation by a global iterative method [26,27].

Suppose each image has *W* rows that are indexed by *j*, and has *H* columns that are indexed by *i*. The pixel is therefore denoted as (*j*, *i*), assuming its depth is *z*(*j*, *i*), and the gradients in the *x* and *y* directions can be expressed as

$$p = \partial z(j, \mathbf{i}) / \partial \mathbf{i}, \ q = \partial z(j, \mathbf{i}) / \partial \mathbf{j} \tag{7}$$

Then,

$$\|m\prime\|\|n\|\| = \left(p\_\prime q\_\prime - 1\right)^\mathrm{T}.\tag{8}$$

The image size is *W* × *H*, so that the discrimination function in the iterative method is

$$E = \frac{1}{W \times H} \iiint \left(\frac{\partial z(j, i)}{\partial i} - p(j, i)\right)^2 + \left(\frac{\partial z(j, i)}{\partial j} - q(j, i)\right)^2 \text{did}j. \tag{9}$$

#### **4. Experimental Results and Discussion**

Existing NPP visual inspection methods are not highly reliable for identifying small defects. This could be improved by using a camera with a higher resolution, which would produce a higher contrast between the defect and the metal surface [1]. However, it can also be difficult to discriminate between true and false defects; therefore, we tested specimens exhibiting small, hard to discern defects.

For the experiments, we used PTZ, introduced in Section 3.2. The layout dimensions of the lens and lights and the image capture setup is shown in Figure 1. With a beam fixing the PTZ, the camera looks downward, and the sample images are captured from the bottom. The four light sources, lasers, and lens are controlled via a controller. In our experiments, the defects were placed immediately below the camera.

Firstly, to show the efficacy of the proposed method for estimating *z*, the estimated distances were compared with the set values, as listed in Table 1, using a distance of 253 mm as the calibration point. The method provides accurate results, ranging from 133–493 mm. The errors were predominantly due to the orthographic image formation model, which does not always precisely reflect the actual conditions. These deviations are also related to the accuracy of laser point extraction.


**Table 1.** Estimation results of the distance between the camera plane and the target plane (mm).

The remaining experiments were conducted with two metal plates, as shown in Figure 4. One contains two small dents, labelled as defect 1 and defect 2 respectively, and the other one contains a pit, labelled as defect 3.

The 3D data generated for each defect are shown in Figures 5–7. The transformation from image pixels to real-world coordinates was achieved as described in Section 3.4. The depth information was then extracted five times along the *Y* direction for each defect, and the maximum depth contour was provided. The 3D data were then combined with altimetric readings of the maximum depth contour for further analysis.

**Figure 4.** Images of test specimens: (**a**) 50 × 20 mm containing two dents (defects 1 and 2); (**b**) size 50 × 60 mm, containing one pit (defect 3).

**Figure 5.** (**a**) Three-dimensional (3D) reconstruction results and (**b**) altimetric readings for the maximum depth in the 3D result for defect 1.

**Figure 6.** (**a**) 3D reconstruction results and (**b**) altimetric readings for the maximum depth in the 3D result for defect 2.

**Figure 7.** (**a**) 3D reconstruction results and (**b**) altimetric readings for the maximum depth in the 3D result for defect 3.

It is clear that the proposed method produces significantly more information than can be observed in the original acquisition images, enabling an accurate characterization of the dimensions and depth of defects. This facilitates image analysis for a range of inspection tasks that could not be solved by 2D analysis of the original acquisition images.

The initial results of the 3D reconstruction show that the proposed method is very useful for identifying defects, and can contribute to more reliable visual inspections with currently available devices. Thus, exploiting available devices will contribute to significant improvements and time savings for NPP visual inspections.

Table 2 shows the evaluation of defect depth. We compare our method with the baseline derived from MarSurf LD 120, which can conduct measurements using the Mahr metrology products. *Z* is the average maximum depth of the five contour lines extracted from the 3D results. The baseline is the average maximum depth of the five contour lines obtained by LD 120, which uses the non-defect area of the metal plate as the reference. LD 120 measures by contact and has a resolution of 0.001 mm in the depth direction, whereas the capacity for our image sensor and lens is approximately 0.030 mm under the proposed setup.


**Table 2.** Comparison of defect depth calculated from the 3D results and obtained by LD 120 [mm].

As shown in Table 2, the depth estimated by our method is close to the baseline. Errors in the experimental setup come from the camera, light sources, object reflectance, etc. The PS technique can deal with these errors to ensure more precise results; however, all PS techniques rely on radiance measurements. The characteristics of the defects, including defect opening displacement and geometry, will affect the validity of the results. In future research, we will determine the parameters and their impacts on the efficacy of depth measurement, which will require further experiments and verification.

Additionally, there are a few limitations of this study. First, it is assumed that the surface reflectance of the object follows the Lambertian model. Figure 6 reveals no highlights or shadows; therefore, there is no distortion in the results. Conversely, Figures 5 and 7 reveal the distortion in specific areas because of specular reflection. The processing of highlights and shadows requires more images, yet the small number of PTZ lights limits highlight and shadow processing. A methodology to handle shadows and highlights with four lights may be useful to improve the performance [28]. It is also important to note that the experimental defects analyzed in this study are more easily processed than the defects observed under actual inspection conditions. That is, the reflectance of the objects does not deviate substantially from the Lambertian model, and the experiment does not include any underwater images. Thus, the next step is to perform extensive experiments under a range of different conditions.

The limitations discussed here do not limit the use of this technique in NPP visual inspections. Moreover, we suggest that this research provides an important basis for developing a method that can readily identify and quantify defects through NPP visual inspection.

#### **5. Conclusions**

This study presents a 3D shape reconstruction method for the visual inspection of defects in NPP reactors. The method is based on the photometric stereo techniques and does not necessitate new inspection devices. The proposed approach, which involves estimating the light source directions and intensities, has reduced the limitation of light calibration and exhibits good practical applicability. The developed methodology can obtain the 3D shape and depth information of defects, thereby improving NPP visual inspection.

The demands for 3D image reconstruction will allow the visual inspection sector to perform more complex and accurate tasks. However, this is only possible if both the software and hardware are improved. The new market applications are expected to continue to emerge as the benefits of a 3D generating function are revealed. This research may also be relevant for designing inspection devices for future generations of NPP reactors.

**Author Contributions:** Conceptualization, K.X.; Data curation, M.W.; Formal analysis, M.W.; Funding acquisition, K.X.; Investigation, S.H.; Methodology, S.H.; Resources, M.L.; Supervision, K.X.; Validation, M.L. and M.W.; Writing—original draft, S.H.; Writing—review & editing, K.X.

**Funding:** This work was funded by National Key R&D Program of China (No. 2018YFB0704304) and the National Natural Science Foundation of China (No. 51674031 and No. 51874022).

**Acknowledgments:** This work was supported by National Key R&D Program of China (No. 2018YFB0704304) and the National Natural Science Foundation of China (No. 51674031 and No. 51874022).

**Conflicts of Interest:** The authors declare no conflicts of interest.
