Image Preprocessing for Outdoor Luminescence Inspection of Large Photovoltaic Parks

Abstract

Electroluminescence (EL) measurements allow one to detect damages and/or defective parts in photovoltaic systems. In principle, it seems possible to predict the complete current/voltage curve from such pictures even automatically. However, such a precise analysis requires image corrections and calibrations, because vignetting and lens distortion cause signal and spatial distortions. Earlier works on crystalline silicon modules used the cell gap joints (CGJ) as calibration pattern. Unfortunately, this procedure fails if the detection of the gaps is not accurate or if the contrast in the images is low. Here, we enhance the automated camera calibration algorithm with a reliable pattern detection and analyze quantitatively the quality of the process. Our method uses an iterative Hough transform to detect line structures and uses three key figures (KF) to separate detected busbars from cell gaps. This method allows a reliable identification of all cell gaps, even in noisy images or if disconnected edges in PV cells exist or potential induced degradation leads to a low contrast between active cell area and background. In our dataset, a subset of 30 EL images (72 cell each) forming grid ($5\times 11$) lead to consistent calibration results. We apply the calibration process to 997 single module EL images of PV modules and evaluate our results with a random subset of 40 images. After lens distortion correction and perspective correction, we analyze the residual deviation between ideal target grid points and the previously detected CGJ after applied distortion and perspective correction. For all of the 2200 control points in the 40 evaluation images, we achieve a deviation of less than or equal to 3 pixels. For 50% of the control points, a deviation of of less than or equal to 1 pixel is reached.

1. Introduction

The photovoltaic (PV) industry has been using luminescence measurements for quality assurance in production lines for several years. It allows the detection of multiple defects like potential induced degradation (PID), defective bypass diodes, cell cracks, disconnected cell parts, busbar soldering defects. Kropp et al. [1] showed that from the defects which are detectable in EL images and their spacial properties, one can conclude to the potential power loss of a PV module and hence to the power loss of PV systems. Alongside the inline luminescence measurement in the production line, the field luminescence measurement for Operation and Maintenance (O&M) [2] purposes is increasingly used since the last years. The DaySy^™ system is an outdoor luminescence measurement system [2] where PV modules do not have to be dismounted. Daylight EL inspection methods like DaySy rely on indium gallium arsenide (InGaAs) cameras with relatively low resolution of less than 1 MP compared to conventional complementary metal-oxide-semiconductor (CMOS) technology.

Luminescence measurements are image-based measurement techniques that use a camera as measuring device. The camera setup including lenses could cause several lens distortion types (e.g., radial, tangential, thin prism). For automated postprocessing procedures, cell extraction, automated defect detection, and the quantitative analysis of luminescence images, these undesired effects must be corrected in advance.

In the field of luminescence measurements Deitsch et al. [3] proposed an automatic segmentation for high resolution electroluminescence (EL) images of PV modules to extract cells and correct radial distortion before. The distortion correction in their method is based on the model of Devernay and Faugeras [4]. The advantage of this approach is that the distortion model estimation is decoupled from the projection to the 2D image plane. Therefore, the intrinsic camera parameters, must not be calibrated. Unfortunately, this method does not consider decentering distortion and is just applicable for EL measurements with minor perspective distortion as explicitly stated in Ref. [3]. For outdoor luminescence imaging of mounted PV modules, where perspective is almost not avoidable, this calibration technique is not applicable.

Bedrich et al. [5,6,7] used the image processing library of OpenCV [8], which is based on the calibration process of Zhang et al. [9] and Heikkilä et al. [10]. This calibration technique usually uses images of a grid of control points extracted from a chessboard as calibration pattern [5,6,7,8,9,11]. It considers radial distortion of multiple orders and decentering distortion. In Ref. [12] Bedrich et al. introduced an automated calibration chain where the cell gap grid of a PV module is extracted from EL images and used as calibration pattern. For the cell gap detection of a PV module in [12], the authors referred to [7], where a vertical and horizontal Sobel–Feldman operator [13] was used to extract edges of the PV cells and then conclude to the cell gaps by two cell edges. As the Sobel–Feldman operator is a gradient based edge detection technique, it has disadvantages in cases of small gradients between active cell area and background of the EL images. Therefore, this edge detection technique may also fail on EL images of PV modules that are heavily affected with potential induced degradation (PID), polarization, or when cell edges are partially or fully disconnected. For the calibration process in Ref. [12], the authors showed a relative line straightness improvement between 5% and 80% for most of the cases. Unfortunately, the results did not show a total accuracy or residual inaccuracy and thus do not allow an overall quantitative quality evaluation of the whole correction process. The authors proved in [12] that in principle it is possible to correct the distortion by detecting the cell gap grid in EL images with minor perspective that are measured in the laboratory. The applicability on EL images of PV modules with high perspective, measured in the field and the the quantitative consistency of the calibration process are not discussed in [12].

Mantel et al. [14] showed an algorithm to detect lines and cell corners in PV modules for perspective transformation by rotating the binary image at different angles and summing the pixel intensity values along the image axes. Their algorithm showed very accurate results for perspective distorted indoor laboratory EL measurements without lens distortion, where only the device under test (DUT) is visible in the EL measurements. Nevertheless their method was less successful for real outdoor EL measurements where noise and neighboring PV modules are present. The applicability of the algorithm on lens distorted EL images is not discussed in their work.

The present work extends the self-calibrating image processing chain (see Figure 1) with a fast, automated, reliable and accurate cell gap joint detection algorithm. The pattern detection is based on a Hough transform [15] applied on an adaptive locally thresholded [16] image. For the first time, an iterative approach based on the linear Hough transform is used to detect the distorted cell gap joints (CGJ) in an EL image. An advantage compared to existing methods is that it is also applicable for real outdoor luminescence measurements in which the PV module is affected by high perspective and large lens distortion. Another advantage of the proposed algorithm is, that it is applicable for luminescence measurements with neighboring PV modules in the image as the segmentation of the DUT is part of the algorithm. The adaptive local threshold allows to extract edges from smaller gradient environment compared to Sobel–Feldman operator as shown in [16]. Compared to the existing methods, this advantage provides more reliable cell gap detection especially for images with smaller gradient between the signal of the active cell area and the background. The detected CGJ in multiple EL images are used as calibration pattern to estimate the distortion and camera coefficients and to correct distortion and the perspective. This work proves that the camera parameter estimation becomes more reliable with more calibration image planes of different scenes. We statistically analyze the amount of images in the calibration subset of our data set necessary to get consistent results, independent of the randomly chosen calibration images. This work also provides a quantitative quality metric to evaluate the accuracy of the image processing chain (lens distortion correction and perspective correction) for the PV module. Compared to Deitsch et al. [3], the algorithm in the present work allows a throughput of 2000 luminescence images in 200 min on an AMD Ryzen^™ 7 2700X 8 core processor which yields a mean time of 6 s per image for the whole image processing chain, including pattern detection, camera parameter estimation, distortion and perspective correction. An advantage of the presented work is that it is applicable for EL images with a lower resolution of ($640\times 512$) px and high perspective of the object in the EL image.

2. Calibration Principle

The automated self-calibration for EL images introduced by Bedrich et al. [12] which is sketched in Figure 1 is an extension on the method of Zhang et al. [9] and Heikkilä et al. [10]. It is based on the pinhole camera model to project a three dimensional (3D) world point (X, Y, Z) into an image plane (Figure 1a–c). The extrinsic camera parameters describe the conversion from 3D world coordinates to 3D camera coordinates. The intrinsic camera parameters describe the projection from 3D camera coordinates to 2D image coordinates u and v. This ideal projection model does not consider lens distortion. Nevertheless, an optical system produces a lens distorted images (Figure 1). The distortion model of Brown [17].

$$\begin{array}{c}\hfill \tilde{u}={\tilde{u}}_d+\underset{\mathrm{radial}}{\underbrace{{\mathring{\tilde{u}}}_d(k_1{r}^2+k_2{r}^4+k_3{r}^6)}}+\underset{\mathrm{decentering}}{\underbrace{\left[p_1(r^2+2{\mathring{\tilde{u}}}_d^2)+2p_2{\mathring{\tilde{u}}}_d{\mathring{\tilde{v}}}_d\right]}}\\ \hfill \tilde{v}={\tilde{v}}_d+\underset{\mathrm{radial}}{\underbrace{{\mathring{\tilde{v}}}_d(k_1{r}^2+k_2{r}^4+k_3{r}^6)}}+\underset{\mathrm{decentering}}{\underbrace{\left[2p_1{\mathring{\tilde{u}}}_d{\mathring{\tilde{v}}}_d+p_2(r^2+2{\mathring{\tilde{v}}}_d^2)\right]}}\end{array}$$

(1)

with ${\mathring{\tilde{u}}}_d=({\tilde{u}}_d-u_0)$, ${\mathring{\tilde{v}}}_d=({\tilde{v}}_d-v_0)$ and $r=\sqrt{{\mathring{\tilde{u}}}_d^2+{\mathring{\tilde{v}}}_d^2}$ describes the correction of a distorted image point $P({\tilde{u}}_d,{\tilde{v}}_d)$ to an undistorted point $P(\tilde{u},\tilde{v})$ with the unknown distortion coefficients $k_1,k_2,k_3,p_1,p_2$. It is assumed as the most complete model because it considers radial distortion of multiple orders and decentring distortion. Other calibration techniques [3,18,19,20,21,22] just consider radial distortion for simplification.

The methods of Zhang et al. [9] and Heikkilä et al. [10] allow the extraction of the extrinsic and intrinsic camera parameters and the distortion coefficients $k_1,k_2,k_3,p_1,p_2$. The algorithms are based on a parameter estimation and non-linear optimization with a source (distorted) and target pattern (undistorted), usually extracted from images of a chessboard pattern. Instead of chessboard pattern, Bedrich et al. [12] used the distorted CGJ pattern of the PV module in Figure 1d) as source pattern, as the square-shaped PV cells form a good regular pattern. A pattern of equidistant grid points in Figure 1e was assumed as target pattern containing the ideal positions of the CGJs. The algorithms of Zhang et al. [9] and Heikkilä et al. [10] estimated the homography between source and target (see Figure 1f) and provide a closed-form solution to get a good initial guess for the extrinsic and intrinsic camera parameter. The distortion coefficients are assumed to be zero for this initial step. With a non-linear optimization, the camera parameters are refined and the distortion coefficients are estimated (see Figure 1g). The abort criterion is predefined by the desired mean reprojection error of all calibration pattern points. The distortion corrected coordinates of a point $P(\tilde{u},\tilde{v})$ in the image are calculated from the distorted coordinates of point $P({\tilde{u}}_d,{\tilde{v}}_d)$ and the estimated camera parameters and distortion coefficients by Equation (1).

As can be seen in Figure 1, a precise and reliable detection of the complete distorted CGJ pattern is absolutely essential for the whole calibration chain because it has strong influence on the parameter estimation. We show a robust pattern recognition algorithm for low resolution, highly distorted outdoor EL images with high perspective, that finds the CGJ between the PV cells of a crystalline silicon module in an EL image (see Figure 1d). The algorithm allows the differentiation between cell gaps and other line structures in the image of a PV module or cell (e.g., busbars) because busbars should be excluded from the calibration pattern.

3. Pattern Detection Procedure

Figure 2 shows the processing steps of the distorted CGJ source pattern detection introduced in this section and discussed in more detail in the following subsections. The subsection structure depicts the structure of the processing steps in Figure 2b–g.

Looking at a crystalline silicon based PV module shows clearly that the cell gaps and busbars form a grid pattern of straight lines. Due to lens distortion, these straight line structures are displayed as curved line structures in an EL image of such PV modules as shown in Figure 2a. First, we find the contours and corner points of the active cell area of the PV module in the center of the image to define a region of interest (ROI) (Figure 2b,c). We assume the PV module in the image center as the object to find the reference pattern and thus isolate this ROI from the rest of the image. This avoids artefacts or other interfering objects (e.g., neighboring PV modules) in the pattern detection process. In the isolated ROI, image binarization and morphological operations are performed to outline the desired cell gap and busbar structures (Figure 2d). The Hough Transform allows the detection of parameterizable geometric forms, such as straight lines and curves in binary images as described in Ref. [15]. For the sake of simplicity we use the Hough Transform in a line search step to find multiple straight line forms in the EL image of a PV module as rough estimation of the curved lines structures in the EL image. This Hough line search step leads to the detection of a cluster of multiple straight lines for one curved line structure (Figure 2e). In a line validation step this cluster is evaluated and filtered for the straight line that represents best the curved line structure. With our line classification algorithm we are able to distinguish between busbar lines and cell gap lines (Figure 2f). The cell gap lines form a regular grid for a rough estimation of the CGJ positions (Figure 2f). As the image is affected by radial and tangential distortion, and thus the CGJ are not at the intersection points of the straight line structures, the CGJ search is refined by using the intersection points of Figure 2f and performing the steps from Figure 2e,f again in a smaller cropped ROI around the intersection point. Figure 2g shows the results before and after the improvement. This results in Figure 2h which shows the refined distorted CGJ positions for the whole PV module. These CGJ are used as reference pattern for the camera parameter and distortion coefficient estimation.

3.1. PV Module Corner Detection

In this section, we describe the corner localization for the module of interest. This is necessary for the following steps in order to provide a robust algorithm. The corner detection process is divided into five parts. In a first step, the solar modules in the image will be segmented and in the following step, the module of interest will be selected. Since the images are taken under real world conditions, the segmented module could suffer from the low image quality and, much more important, some corners could lie outside of the image area. Therefore, a robust estimate of the module contour is needed in order to get the exact corner coordinates. To achieve the precise estimate, we assume a geometric model for the PV modules and fit the segmented contour in a robust way to the assumed model.

To segment the module, we first apply a bilateral filter to reduce image noise without smoothing out the edges of the module contour [23]. To distinguish between pixels belonging to the PV module and pixels belonging to the background of the EL image, we use either an Otsu Thresholding which is based on the gray-level histogram of the image [24] or an adaptive local median thresholding algorithm [16]. Since the pixel intensity is inhomogeneous, the thresholding does not lead to the desired segmented module. Therefore, we apply erosion and dilation to the thresholded image in order to isolate individual pixels and joining separate pixels in the EL image [25]. An example of a segmented module is shown in Figure 2b). For further processing, the contour of the segmented module area is needed. Therefore, we use a common contour finding algorithm [26]. In a last step, the contour of the module of interest is selected based on its moment. The contour with the closest moment location to the image center will be selected. The result is shown in Figure 2c).

As already mentioned, the extracted contours often suffer from the low image resolution which results in noisy contour points. Furthermore, it is possible that some parts of the module lie outside of the captured image. To overcome those two severe problems, we suggest to robustly fit the module contour to a geometric model. A simple approach would assume the rectangular geometry of the real PV module. Due to the heavy lens distortion of the used cameras, this assumption is not suitable. We suggest using a geometric model consisting of four parabolic line segments. To easily fit this geometric model, we group the contour points into four groups for each edge and afterwards fit each group to a parabolic model. To split the contour points, we first calculate the distance for each contour point to the PV module center. The maximum distances are assumed as a rough estimate of the corners. By using the estimated corners, we select the corresponding contour points for each edge of the module. To robustly fit each parabolic model, we suggest using the RANSAC algorithm [27]. To get the coordinates of the module corners in a last step, the intersection points of the neighboring parabolic functions have to be calculated.

3.2. Binarization for Cell Gap Detection

Figure 2d shows the binarization for the cell gap detection. The detected corners of Section 3.1 in Figure 2c define a ROI for the cell gap detection. In EL measurement only active cell area emits a radiation signal that is detected with the camera system. Background, cell caps, busbars, disconnected cell areas and cracks do not emit a radiation signal and therefore appear dark in a greyscale EL image (see Figure 2a. An adaptive local threshold [16] with a kernel size of 5 × 5 is applied to the ROI window to binarize the image in the ROI. Every pixel outside the ROI is set to zero (black). The pixel values in the binarized ROI are inverted to emphasise the cell caps, busbars, disconnected cell areas and cracks instead of the active cell area as can be seen in Figure 2d. Morphological operations (erosion and dilation) remove artefacts and close small gaps in the structures.

3.3. Identification of Line Structures

To detect line structures in the inversed binarized image in Figure 2d), the perspective position of the PV module is estimated from the cornerpoints of Figure 2c). From the perspective position and the amount of cells in the PV module we define ROIs from the long-to-long module edges and from short-to-short module edges. These are potential locations of line structures that represent cell gaps. A line detection algorithm based on Hough transformation [15] in the entire image and in each specific ROI results in a set of multiple straight lines which is shown in Figure 2e. In order to get the best representation of a cell gap line and distinguish between busbar and cell gap we propose a three step validation process:

• Filter out straight lines by angle and position that do not match the perspective of the PV module
• Reduce the number of straight lines for one curved line structure to one straight line with the closest representation
• Distinguish between cell gap line and busbar line

Depending on the perspective of the PV module in the image, only straight lines in a certain angular range and position are valid and are clustered for short-to-short edge lines and long-to-long edge lines. The following characteristics are used to reduce the set of lines to the best representation of one straight line for each curved line structure (cell gap lines or busbar lines):

• Weight ($w_i$) of each line—sum of distance weighted pixel intensity.
• (Intersection) of lines. Wherever two found lines of the same cluster (long-to-long or short-to-short module edge) intersect, the line with the higher weight is assumed as the valid line and is used for further steps.

The validation process results are shown in Figure 2f. For each line structure in Figure 2f there exists exactly one found straight line that represents best the curved line structure in the binary image. Let

$$g_{x,y}:\overrightarrow{x}=\overrightarrow{a}+\lambda \overrightarrow{b}$$

(2)

be the parametric representation of a valid found line in long-to-long or short-to-short module edge direction in Figure 2f, where $\overrightarrow{a}$ is the position vector, $\overrightarrow{b}$ the directional vector and $\lambda$ is the parameter. As mentioned before, the valid found straight lines $g_{x,y}$ are rough approximations for both busbars as well as cell gaps in Figure 2f. For an equidistant calibration pattern or for location of PV cell borders one has to distinguish between cell gap line structures and busbar line structures.

By knowing the PV module corner points from Section 3.1 and the perspective, we now approximate a distortion free grid between the PV module corner points that represents the cell gap grid pattern and hence the perspective.

Assume the matrix

$$M=\left(\begin{array}{ccccccc}{P}_{11}& P_{12}& P_{13}& ...& P_{17}& P_{18}& P_{19}\\ P_{21}& P_{22}& P_{23}& ...& P_{27}& P_{28}& P_{29}\\ P_{31}& P_{32}& P_{33}& ...& P_{37}& P_{38}& P_{39}\\ P_{41}& P_{42}& P_{43}& ...& P_{47}& P_{48}& P_{49}\\ P_{51}& P_{52}& P_{53}& ...& P_{57}& P_{58}& P_{59}\end{array}\right)$$

(3)

to be the matrix with the 45 grid points $P_{mn}$ in the approximated grid for a 60 cell PV module in landscape format with $1\le m\le 5$ and $1\le n\le 9$.

The points $P_{m1}$ to $P_{m9}$ represent an imaginary line $h_m$ from the short edge on one side to the short edge on the other side of the PV module. The points $P_{1n}$ to $P_{5n}$ represent an imaginary line $h_n$ from long to long edge of the PV module.

For each point P on the imaginary lines $h_m$ and $h_n$ the distance

$$d_{p_{mn}}=\frac{|({\overrightarrow{p}}_{mn}-\overrightarrow{a})\times \overrightarrow{b}|}{|\overrightarrow{b}|}$$

(4)

represents the distance of this point to a valid straight line of the same cluster (short-to-short edge line or long-to-long edge line) in Figure 2f.

The mean distance

$${\overline{d}}_{h_m}=\frac{1}{9}\sum _{n=1}^9{d}_{p_{mn}}$$

(5)

respectively

$${\overline{d}}_{h_n}=\frac{1}{5}\sum _{m=1}^5{d}_{p_{mn}}$$

(6)

is assumed as mean distance between an imaginary grid line and a valid straight line of its cluster.

For each imaginary line $h_m$ or $h_n$, its four nearest valid straight lines $g_i$ of the same cluster, given by the four smallest mean distances ${\overline{d}}_{h_m}$, respectively, ${\overline{d}}_{h_n}$ define a set $G=\{g_1...g_4\}$. If two imaginary lines $h_m$ and $h_{m+1}$ (or $h_n$ and $h_{n+1}$) share the same line $g_i$ in their set, the line is assigned to the set with the smaller mean distance ${\overline{d}}_{h_m}$ (or ${\overline{d}}_{h_n}$). This results in sets $G=\{g_1...g_i\}$ ($1\le i\le 4$) with a minimum size of two and a maximum size of four nearest valid straight lines (cell gap lines and busbar lines) for each imaginary line.

Further for each line $h_m$ or $h_n$, the line in G most likely to be a cell gap line is searched.

Three different key figures (KF) are defined to determine the cell gap line:

For the ${\mathit{KF}}_{{\mathit{i}}_1}$, all lines in G are sorted by their weight $w_i$, which is defined earlier in this section. The key figure

$$K{F}_{i_1}=\frac{w_i}{w_2}-1$$

(7)

compares each line in the sorted set G with the line that has the second highest $w_i$.

This results in positive $K{F}_{i_1}$ values for strong and long line structures in G and negative values for thinner and shorter line structures.

Let $Q_i$ be the intersection point of a line $g_i\in G$ with the vertical perpendicular bisector of the bounding box around the PV module area in the image. The set $S=\{{\overrightarrow{q}}_1,...,{\overrightarrow{q}}_i\}$ defines the set of the position vectors of the intersection points of all lines $g_i\in G$ with the vertical perpendicular bisector.

The distance

$$d_{ij}=|{\overrightarrow{q}}_i-{\overrightarrow{q}}_j|$$

(8)

is the distance between two intersection points in S and is assumed as distance between two lines in G. The set

$$D_i=\{|{\overrightarrow{q}}_i-{\overrightarrow{q}}_j|\mid {\overrightarrow{q}}_i,{\overrightarrow{q}}_j\in S,j\ne i\}$$

(9)

contains the distances of a line $g_i\in G$ to the other lines in the set. The union set

$$D=\stackrel{.}{\bigcup _{i\in I}}{D}_i$$

(10)

contains the distances of all lines in a set G to each other.

The ratio

$$K{F}_{i_2}=\frac{min\left(D\right)}{min\left(D_i\right)}$$

(11)

is defined as second indicator for each $g_i\in G$. Lines that are close to other lines in the set have a higher ${\mathit{KF}}_{{\mathit{i}}_2}$.

The ratio

$$K{F}_{i_3}=\frac{min\left(D\right)}{max\left(D_i\right)}$$

(12)

is defined as third indicator for each $g_i\in G$. Lines that are located in the center of a set have a higher ${\mathit{KF}}_{{\mathit{i}}_3}$.

All indicators yield to a total indicator by the weighting function

$$K{F}_{i_{total}}=2K{F}_{i_1}+1K{F}_{i_2}+1K{F}_{i_3}.$$

(13)

The line with the highest total key figure ${\mathit{KF}}_{{\mathit{i}}_{\mathit{total}}}$ is considered most likely to be the line that represents the cell gap line. This separation technique considers the main features which distinguish a cell gap line from a busbar line in the image and can be adjusted individually for different PV module types.

3.4. Precise Cell Gap Joint (CGJ) Search in ROI

The intersection points of the cell gap lines in Figure 2f do already give a precise representation of the cell gap joints in the center of the PV module. The location of the intersection points near the PV module corners do not match exactly due to the lens distortion as can be seen in Figure 2g. Additional iteration steps of the Hough line algorithm, validation and classification steps in individual variable windows around the CGJs refine the location of each CGJ, which is shown in Figure 2g. This individual refined CGJ pattern shown in Figure 2h is a good approximation for the distorted source pattern shown in Figure 1d.

4. Experiment

4.1. Setup

In the following experiments, a dataset of 997 EL measurement images of PV modules is used, that are measured in the field by the Solarzentrum Stuttgart GmbH. Each EL measurement image displays one complete PV module and parts of neighbouring PV modules as can be seen in Figure A1. The EL images in the dataset show different PV modules but from the same manufacturer and PV module type. All images in the sample set are measured with the same objective (focal length f = 25 mm). The system includes a 1,3 MP camera with an InGAs camera sensor and a resolution of (640 × 512) px. The DaySy^™ system provides two outputs as can be seen in Figure A1 and Figure A2. Figure A1 shows a typical DaySy^™ system EL measurement image of a defective multicrystalline silicon PV module with 72 PV cells. Figure A2 shows a photography image of the same scene as Figure A1. Therefore both images are highly affected by the same perspective and lens distortion and both are used as input images for the pattern detection algorithm. In case that the detection of the distorted CGJ pattern in the EL image is not accurate, alternatively the photography image is passed through the same algorithm.

The PV modules in the EL images contain 72 PV cells ($6\times 12$) that allow one to extract a pattern of 55 CGJ points ($5\times 11$). Any other calibration pattern for the optical system is not available and the optical system is not calibrated in advance.

4.2. Results

Two experiments are performed to evaluate the presented pattern detection algorithm and the distortion correction acc. to Bedrich et al. [12], Zhang et al. [9] and Heikillä [10] for highly perspective and highly lens distorted images.

4.2.1. Experiment 1—Parameter Estimation Quality Depending on the Amount of Image Planes

This experiment investigates the parameter estimation quality of the intrinsic camera parameters $f_x$ and $f_y$ (focal length camera parameters) and $u_0$ and $v_0$ (principal point camera parameters) in Figure 1b) depending on the amount of different PV module planes used for calibration. First, we select manually 20 EL images of PV modules with various orientations from our 997 EL image dataset. These 20 EL images are used as calibration subset. Zhang et al. showed in Ref. [9] that images of the calibration pattern measured from at least three different perspectives are essential to estimate all intrinsic parameters. The parameter estimation from multiple images acc. to Zhang et al. uses a nonlinear optimization as described in detail in Ref. [9].

Each time a parameter estimation is performed from a different random sample set (here 55 CGJ points), the estimated parameter slightly differs from the true parameter of the model. Performing this parameter estimation multiple times leads to a distribution of this estimated parameter. The standard error

$$SE=\sqrt{\frac{{\hat{\sigma }}^2}{N}}$$

(14)

gives a quality criterion for the parameter estimation, where ${\hat{\sigma }}^2=s^2\frac{N}{N-1}$ is the estimated variance of the population, calculated from the sample variance $s^2$ and the correction term with the sample size N Let $\widehat{y}$ be the mean value of the estimated parameter. The relative standard error (RSE)

$$RSE=\frac{SE}{\widehat{y}}$$

(15)

is the Standard Error (SE) divided by the estimate $\widehat{y}$.

Figure 3 shows the relative standard error (RSE) for the camera parameters ($f_x$, $f_y$, $u_0$, $v_0$) for the n first images in the calibration subset from $n=3$ to $n=20$. It is assumed that the pixel of the camera chip are square shaped and thus the pixel skew coefficient c is neglected.

The RSE decreases with the amount of image planes used for the parameter estimation and therefore the approximation becomes more significant. The decreasing RSE is based on the decreasing SE with increasing sample size and decreasing variance. This is similar to the results of Zhang et al. [28]. Just three to four images are sufficient to get a RSE below 2% for all the parameters in Figure 3. With 10 images or more the RSE of the parameters in Figure 3 falls below 1%.

Figure 4 shows the RSE for the radial distortion parameter $k_1$ for the n first images in the calibration subset from $n=3$ to $n=20$.

Similar to Figure 3, the RSE decreases with the number of image planes used for the parameter estimation. At least seven images should be used for this calibration subset to get a reliable RSE below 20%. With all 20 calibration images in the subset, a RSE 10.8% is achieved for the radial distortion parameter $k_1$.

The result of this experiment is highly dependent on the manually selected EL images especially the different PV module plane angles and the different formats of the PV module in the EL image. In an automated self-calibration preprocessing of EL images for field and industry application it is useful to not choose the calibration images manually. In this case, a proper solution is to random select the calibration image subset out of the complete dataset.

4.2.2. Experiment 2—Calibration Quality for Large On-Field Datasets

This experiment investigates the application of the pattern detection and camera calibration of Figure 1 and Figure 2 for a large dataset of EL images to prove the applicability on a typical EL field measurement service with many EL images. A randomly selected subset of calibration images is used to perform pattern detection and camera calibration. For the dataset in this work, 20 calibration rounds with different amounts of random selected calibration images are performed. Figure 5 shows the statistical distribution of the RSE after 20 rounds.

The random selection of just a few calibration images leads to a high variation of the standard error of the camera parameter estimation. This is due to a low variation in the perspectives of the PV module planes in this EL image dataset. Nevertheless, a small set of 10 images is still sufficient to get a reliable RSE smaller than 8% in 19 of 20 rounds for $u_0$ and smaller than 2% for all rounds for the other parameters. For the dataset in this work, a sample size of 30 randomly selected EL images lead to a relative standard error below 1% for all intrinsic camera parameters $f_x$, $f_y$, $u_0$, $v_0$, for all 20 calibration rounds.

Quality Evaluation Metric

For the evaluation of the self-calibrating image processing chain, the pattern is detected acc. to Section 3 followed by a distortion correction acc. to Figure 1 introduced by Bedrich et al. [12]. After distortion correction, the CGJ search algorithm is performed again and the image is perspective transformed by knowing the CGJ points. Compared to existing methods, where the four corner points of the PV module are used, we use all CGJ points and thus more reference points for the perspective transformation. More available reference points, e.g., 45 reference points in a 60 cell PV module, also allow more complex geometrical transformations such as polynomial transformations of higher order. As an example, Appendix A shows three images before, after distortion correction and after perspective transformation. After distortion correction and perspective transformation we use the following three control patterns to evaluate the accuracy of the image processing chain:

• Ideal equidistant grid points, in a undistorted and perspective transformed image—target points (of perspective transformation)
• CGJ points as detected in the distorted image, undistorted and perspective transformed—source
• Ground truth CGJ points labeled by an expert in the distortion corrected and perspective transformed image—real location of CGJ after correction

We define the inaccuracy of the distortion correction and perspective correction process by the 2D euclidean distance

$$d\left(p,q\right)=\sqrt{{\left(u_q-u_p\right)}^2+{\left(v_q-v_p\right)}^2}$$

(16)

between two corresponding CGJ control points $p(u_p,v_p)$ and $q(u_q,v_q)$ of two different control patterns. A small Euclidean distance between the expert labeled ground truth CGJ points and algorithm detected CGJ points, signifies a successful CGJ pattern recognition algorithm which leads to a good distortion correction. A small Euclidean distance between the expert labeled ground truth CGJ points and the target pattern CGJ points, shows the success of both distortion correction and perspective transformation. A small Euclidean distance between the algorithm detected CGJ points and the target pattern also indicates success of distortion correction since the algorithm detected CGJ points are therefore aligned on straight lines in both the distortion corrected and perspective corrected image. Figure 6 shows an image detail of four cells of a PV module EL image after the preprocessing steps of Figure 1 followed by perspective transformation.

Figure 6 shows that the CGJ control points do not always match exactly because of the residual inaccuracy of the distortion correction and perspective transformation. Nevertheless, Figure 6 indicates that a high accuracy is achieved. A Euclidean distance of 0 px describes an exact match of the control points. A Euclidean distance of 1.41 px describes a deviation of 1 px in each direction (u and v). A Euclidean distance of 2.89 px describes a deviation of 2 px in each direction (u and v).

Self Calibration Quality

This experiment shows the statistical distribution of the Euclidean distances between the algorithm found control points and target pattern points for the f = 25 mm objective in the dataset. The statistical data consists of 997 EL images of 997 different PV modules (72 cells each) (see Figure A1) which results in 54,835 algorithm detected control points. For each algorithm detected control point, the Euclidean distance to the corresponding target pattern point is calculated by Equation (16). Figure 7 shows the statistical distribution of the Euclidean distances.

As can be seen in the box plot in Figure 7, 75% of the Euclidean distances are below 1 px difference. In 99% (whisker length) of the cases, the Euclidean distances are below 2.24 px difference, which is less than ±2 px difference between ideal point position and perspective corrected position of the algorithm detected CGJ point in the distortion corrected image.

For the evaluation with the ground truth CGJ points the CGJ points in 40 random selected EL images were labeled, which provides 2200 ground truth control points in total. Figure 8 shows the results of the Euclidean distances between all 2200 control points of the ground truth dataset and the corresponding algorithm detected CGJ pattern (after distortion correction and perspective transformation) and the ideal target pattern points. A comparison of the Euclidean distances between the algorithm detected CGJ points and the ideal target pattern points in Figure 8 with the results in Figure 7 for the whole dataset (54,835 control points) shows that the results are very similar. This allows the conclusion that the results for the 2200 control points (40 ground truth images) are statistically significant for the whole dataset.

The Euclidean distances between the algorithm detected control points (after distortion correction and perspective transformation) and the ground truth control points in Figure 8 show a larger displacement than the distances of the algorithm detected CGJ points and the ideal fix grid control points. The exact position (Euclidean distance below 1 px) is achieved for 25% of the control points. Figure 8 shows 50% of the Euclidean distances are below 1.41 px and 75% below 2.24 px. Almost all control points distances (99%) are below 3.16 px, which is a deviation of 1 px in one image coordinate direction and 3 px in the other image coordinate direction. The absolute maximum value of the Euclidean distances in this dataset is 4.24 px which represents a deviation of 3 px in both coordinate directions and thus a maximum inaccuracy of 3 px in pattern detection. Looking at the deviation between ground truth pattern and the ideal fix target pattern, 99% of the control points are within a range of less or equal to 2 px, which is represented by the Euclidean distance of 2.82 px.

If we consider, that the cell gaps in this EL images have a width of 4–5 px in the sharp image areas, the results let conclude that the pattern detection and distortion correction is really accurate and precise and we can achieve a almost perfect perspective transformation. Appendix A shows the EL images before and after lens distortion correction and the final output image after perspective correction that is used for further defect impact analysis algorithms.

5. Discussion

The proposed pattern recognition algorithm and the calibration chain excellently works with highly distorted low resolution EL images of crystalline silicon based PV modules measured under field conditions. No other camera calibration pattern is required. A high accuracy and precision for the camera parameter estimation is achieved and a generalization for a randomly selected calibration subset is possible. A premise for the algorithm is the knowledge of the cell geometry in the PV module that is measured. It is also a prerequisite, that the PV module under test shows a detectable regular cell gap pattern with cell gap joints. The proposed algorithm is not applicable for standard thin film PV modules, because they show only cell gaps in one module direction and therefore, do not show cell gap joints. PV modules with shingled PV cells are not tested in this work but could possibly lead to an irregular cell gap pattern. Standard half-cut PV cell modules usually show a regular cell gap joint pattern that is detectable but could lead to a more unequal distribution of calibration points (e.g., $5\times 23$) and thus could cause a higher difference of the relative standard error of the estimated camera parameters when just using EL images of PV module with the similar orientation.

For the random sample dataset in this work, a minimum of 30 calibration images leads to a RSE below 1% for the camera parameter estimation; a high precision is achieved. Using more calibration images results in a more stable camera parameter and distortion coefficient estimation, which is shown in the smaller distribution of the standard error of the estimated parameters in Figure 4 and Figure 5. For a calibration with 30 randomly selected images, a high calibration accuracy of less than ±2 px for 99% of the detected CGJ points compared to the ideal fix grid point position is achieved. All of the 2200 ground truth points labeled by an expert after distortion correction and perspective transformation are within a precision range of ±3 px from the ideal expected target point position, which shows that the camera calibration and perspective transformation are accurate. The algorithm in this work is designed for low resolution real outdoor EL images with a fixed images size of 640 px × 512 px. For other image sizes, the filter kernel sizes as well as the ROI window sizes for the detailed line search should be adjusted. The proposed pattern recognition algorithm and camera calibration chain seems also applicable for on-field photoluminescence images.

6. Conclusions

This work proposes a robust pattern recognition algorithm for detecting cell gap joints in EL images of PV modules. The detected cell gap joints pattern is used as calibration pattern for a camera calibration acc. to Bedrich et al. [12], which is based on the algorithm of Zhang et al. [9] and Heikkilä et al. [10]. For EL images with a resolution of 640 px × 512 px, the pattern recognition algorithm results in a camera calibration of high accuracy and precision of ±2 px difference between ideal point position and perspective corrected position of the algorithm detected CGJ point in the distortion corrected image for 99% of the control points. The more calibration images are used, the more stable is the camera parameter and distortion coefficient estimation. The distortion corrected and perspective transformed EL images of the PV modules are used for further image processing steps such as defect localisation and measurements in PV cells and PV modules, e.g., by machine learning (ML) algorithms. The pattern detection algorithm allows also the extraction of individual PV cells in the image. For cell based ML, this could improve an automated preprocessing of the training data.

For online EL monitoring, e.g., with drones in the future, a calibration of the camera setup is not necessary before starting the measurement. The operator can start to capture EL images under different perspectives. The EL images can be from different PV modules in the PV-Park or from the same module at different angles. The algorithm automatically detects the CGJ pattern and calculates the camera parameters and lens distortion coefficients while the measurement continues. When the desired quality acc. to Section 4.2.2 is achieved, the drone system can store the lens distortion coefficients and then correct the lens distortion in all images without a new estimation. The lens distortion correction with this distortion coefficients is allowed until the camera-objective setup changes. Afterwards the algorithm only has to detect the CGJ pattern in undistorted images and make a simple perspective transformation to display the DUT.

As the CGJ pattern detection algorithm is very fast, easy computable, accurate and works with both on-field EL images and photography images of tilted PV modules, it could be also useful for other drone based applications. From the estimated intrinsic and extrinsic camera parameters of the camera model used in this work, it is not only possible to estimate lens distortion coefficients, it is also possible to determine the location of the camera and the DUT in the 3D space, which could be useful for object tracking, distance measurement to the object under test or orientation measurement as well as for focusing applications. For this “real-time” online applications, the software should be optimized in the future to improve processing performance. In an EL image of a standard crystalline silicon based PV module the detection of the 45 CGJ points in a 60 cell PV module (or 55 CGJ points in a 72 cell PV module) also allows more complex geometrical transformations (e.g., polynomial transformation of higher order) which could lead to a more precise correction that is useful for image registration.