A CNN-Architecture-Based Photovoltaic Cell Fault Classification Method Using Thermographic Images

Abstract

Photovoltaic (PV) cells are a major part of solar power stations, and the inevitable faults of a cell affect its work efficiency and the safety of the power station. During manufacturing and service, it is necessary to carry out fault detection and classification. A convolutional-neural-network (CNN)-architecture-based PV cell fault classification method is proposed and trained on an infrared image data set. In order to overcome the problem of the original dataset’s scarcity, an offline data augmentation method is adopted to improve the generalization ability of the network. During the experiment, the effectiveness of the proposed model is evaluated by quantifying the obtained results with four deep learning models through evaluation indicators. The fault classification accuracy of the CNN model proposed here has been drawn by the experiment and reaches 97.42%, and it is superior to that of the models of AlexNet, VGG 16, ResNet 18 and existing models. In addition, the proposed model has faster calculation, prediction speed and the highest accuracy. This method can well-identify and classify PV cell faults and has high application potential in automatic fault identification and classification.

1. Introduction

In recent years, facing the problems of carbon emissions and environmental protection, new energy such as PV power generation has attracted more and more attention. Moreover, global dependence on the use of electrical energy is increasing rapidly, which also increases people’s interest in developing renewable energy [1]. Among a series of renewable energy sources, such as solar, wind, hydroelectric and geothermal energy, wind, hydroelectric and geothermal energy are mostly limited by regions, seasons and climates, and their utilization and production rates are low. On the contrary, solar energy systems play a vital role in providing stable power demand due to their advantages of being environmentally friendly, low-carbon energy, with safe operation, noiseless impact and low cost [2]. The International Renewable Energy Agency reported that the global solar PV net capacity additions from 2019 to 2021 were 109.6 GW, 134.9 GW and 151 GW, respectively [3]. In PV systems, various anomalies usually lead to losses of electricity production and affect the working efficiency and even the operational safety of modules. These anomalies may be caused by harsh weather in actual use, or mechanical damage during manufacturing, installation and transportation. These damages will also shorten the actual service life of PV modules [4]. Therefore, it is still a problem that needs to be solved to effectively monitor the health of PV cells and ensure their performance and safety [5].

Several methods, such as EC (Electrical Characterization) [6], EL (Electroluminescence) imaging [7] and infrared imaging [8] methods have been adopted to detect defects in PV cells. The principle of EC is to find out the faults of PV cells by analyzing the I-V characteristics, but some small faults will hardly affect the characteristics, and this method cannot locate the fault area. In contrast, the methods of EL and infrared imaging technology more easily recognize and locate faults. By obtaining the gray image of the tested object in the dark, EL imaging can effectively identify the existing micro-crack fault. However, due to the single image color, fault identification is time-consuming and expensive, so EL imaging is only suitable for small-scale fault detection. The infrared imaging method uses different colors to represent the temperature distribution of the tested object, so as to judge the existence form of faults. Compared with EL imaging, infrared imaging is more suitable for large-scale fault detection.

For large-scale applications, several infrared thermography (IRT) methods linked to passive [9] and active [10] approaches have been applied to detect various faults in PV cells. He et al. [11] used electromagnetic induction infrared thermography (EIIT) to detect scratches, hot spots and other faults in PV cells, and analyzed the thermography sequence through Fast Fourier Transform (FFT), Independent Component Analysis (ICA) and Principal Component Analysis (PCA). Breitenstein et al. [12] used dark lock-in thermography (DLIT) technology to analyze the leakage current phenomenon, local efficiency and I-V characteristics of PV cells. In particular, he used DLIT to evaluate short circuit current density imaging, and experimental results showed that this method can improve the accuracy of DLIT-based local efficiency analysis of solid cells [13]. In another study, the “Local I-V” method for evaluating DLIT images was applied for high efficiency monocrystalline silicon solar cells, and the results indicate that DLIT can be used to image and quantify the local inhomogeneous dark current contribution of PV cells [14]. Straube et al. [15] used illuminated lock-in thermography (ILIT) to analyze the shunt phenomenon in PV cells. The efficiency of DLIT and ILIT is compared by Frühauf et al. [16]. With the increasing demand for large-scale production quantities of PV cells, fault detection needs to change from traditional visual detection to automatic detection. In particular, the application of AI (artificial intelligence) algorithms and deep learning makes fault detection and classification of PV cells’ thermography images more efficient. Akram et al. [17] proposed an isolated-learning-model-based automatic defects detection method for PV cells using infrared images. A light CNN was designed to train the isolated learning model from scratch with an accuracy of 98.67%. When developing the model transfer deep learning method, a base model was first pre-trained using an EL image dataset of PV cells. The next step was fine-tuned training on an infrared image dataset, which achieved an accuracy of 99.23%. However, the time cost of this model takes 1 h and 17 min, and the research focuses on the datasets of two types of faults. Chen et al. [18] proposed a visual fault detection method with a multi-spectral deep convolutional neural network (CNN) to analyze the light spectrum features of PV cell color images. The method can detect existing surface faults with an accuracy of 90%. However, the proposed multi-spectral CNN has weak feature extraction capabilities for small defects such as cracks. Deitsch et al. [19] used CNN and SVM to detect PV cell EL images. The results indicate that the CNN model has a higher accuracy than SVM, with an average accuracy of 88.42%. However, the trained CNN model requires higher computational power and consumes a large amount of resources. Tang et al. [20] used an image generation method and a CNN-model-based PV cell faults classification using electroluminescence images for the samples. The image generation method combines traditional image processing and Generative Adversarial Network (GAN) characteristics. The accuracy of the proposed CNN model was about 80% for micro-crack, fault-free and break-fault types, respectively. However, this study mainly inspected the same type of defects, such as micro-cracks and finger interruption and breaks. Akram et al. [21] used a light convolutional neural network model based identify fault in EL images of PV cells, and various data augmentation methods were evaluated to solve data scarcity. The model obtained an ideal detection result of 93.02%, and it took only 8.07 ms to predict one image. Cipriani et al. [22] proposed a novel method based on CNN to realize the classification of dust and hotspot faults in PV systems through thermography technology. The experimental results achieved an accuracy of 98%. However, the dataset used in this study was too sparse. Wang et al. [23] proposed a hybrid algorithm by combining the symmetrized dot pattern (SDP) with a convolutional neural network (CNN) for PV module fault recognition. Three faults such as poor welding, breakage and bypass diode were discussed. The experimental results show that the proposed algorithm can capture the fault signals effectively, display them in images and recognize the PV modules’ fault types accurately. The literature review indicates that although deep learning has shown some performance in EL studies, existing studies still have limitations such as high computational cost, low performance and detection of specific defects, etc. In order to better overcome these limitations and ensure the healthy performance of PV cells, it is possible to achieve efficient and low-cost PV fault detection in infrared images by adopting more suitable CNN models and generalization strategies. Therefore, we integrated residual structural units in the series network model and propose a CNN model based on infrared image features of PV cells to achieve automatic classification of cell faults and predict their power generation efficiency and potential safety issues.

In summary, the current research mainly analyzes and obtains the existing faults through the EL images of PV cells in outdoor service, while there is less research on the detection and classification of faults that may occur during the manufacturing process of cells. This process requires faster detection efficiency and fault classification speed. A new CNN model is proposed to process the cells’ infrared image dataset to meet this requirement. First of all, infrared images of PV cells with faults were collected, and offline data augmentation technology was adopted to solve the problem of data scarcity and improve the generalization ability of the network. Secondly, the designed CNN model integrated the residual units, which can fully extract the deep features of PV cells. Finally, the proposed model was verified on the pre-treated infrared PV cells dataset, and the results show that the proposed model can better perform the task of PV cell fault classification.

The remainder of our paper is organized as follows. First, data materials and methods with dataset description, data augmentation and the CNN architecture of the adopted method are given in Section 2. Subsequent to this, the experiments, evaluation criteria and findings results and discussion are given in Section 3. Finally, the conclusion is provided in Section 4.

2. Data Materials and Methods

2.1. Experimental System of the PV Cell Fault Detection

The PV cell fault detection schematic diagram is shown in Figure 1a and the experimental system platform built is shown in Figure 1b. The programmable power supply was modulated by a Data Acquisition (DAQ) card and provided pulse forward bias voltage as the electric excitation. An uncooled infrared camera was applied to capture the thermography sequence. With the purpose of ensuring the synchronization of image acquisition and power excitation, the trigger signal for infrared camera image acquisition and the switch signal of programmable power supply were controlled by the DAQ card [24].

The PV cells were excited with a forward bias voltage produced by programmable power supply, the cells generated a diffusion current and a depletion region recombination current. Some defects in the photovoltaic cell acted as recombination centers by introducing different energy levels in the band gap; the heat dissipation and transport mechanisms of the photovoltaic cells were affected, which shows that part of the current was converted into heat and continued to accumulate, thus affecting the surface temperature distribution of the photovoltaic cells. We captured the surface temperature information of photovoltaic cells with an infrared camera, and converted it into visual thermography sequences for subsequent processing and analysis. The flowchart analysis process is shown in Figure 2.

2.2. Infrared PV Cells Dataset

The infrared image dataset contains 5026 images with a resolution of 100 × 100. There are two fault classes: hot spot and crack, and one no-fault class. The number of each class in the infrared PV cells dataset is shown in

Table 1. Number of each class in the infrared PV cells dataset.

Class Name	Number of Images
	Original Dataset	Offline Data Augmentation
Cracking	161	1127
Hot spot	189	1323
No-fault	368	2576
Total	718	5026

, and the sample images randomly selected from the dataset are shown in Figure 3. For example, cracking appeared in different positions on PV cells, showing different lengths, straight lines or curves. Most of them are generated in the manufacturing process, and the possibility of crack growth will increase during the subsequent welding process. In thermal images, the hot spot is often shown in the form of high-temperature areas, because these areas have been completely transformed into heat. In addition, the number of samples for the no-fault class accounts for 50% of the total sample data, which is the result of offline data augmentation. The offline data augmentation method will be described in the following section.

2.3. Offline Data Augmentation

When using CNN for classification tasks, the number of the dataset affects the convergence performance and generalization capability of the model. There are 718 images in the original dataset in this study, and the total number is small. Therefore, an offline data augmentation method needs to solve data scarcity [25]. To ensure the full effectiveness of the dataset, three methods were selected here: rotation, reversing and contrast enhancement. The rotation operation was to rotate 90, 180 and 270 degrees around the center of the image. The reversing operation was to reverse the pixels in each row and column of the images, so two images could be obtained. Contrast enhancement was achieved by improving the local contrast of the images. Figure 4 shows the example results of each of the above operations. After offline data augmentation, the total number of the expanded dataset is 5026 infrared images.

2.4. CNN Architecture of the Proposed Method

As shown in Figure 5, the structure of the proposed PV cell fault classification method includes two processing processes: a data pre-processing process and a network training and validation process. In the pre-processing, offline data augmentation is used to improve the network performance and retain the original image tag while effectively expanding the dataset. In network training and validation, a CNN architecture integrating residual units is constructed for training and testing. The residual learning can optimize the convolutional layer in front, protect the information in the middle layer when simplifying the training of the deep neural network (DNN) and can sufficiently extract the salient features.

The proposed model includes 6 convolutional layers, 3 max-pooling layers, 3 fully connected layers and 1 classification layer. Among them, the Res-Model contains 3 convolutional layers.

Table 2. Details of the proposed CNN network.

Layer Type	Parameter Settings
Conv-1	64 (11 × 11) filters with stride 1 [ReLU, L2]
Max-pooling	2 × 2 filter with stride 2
BN	-
Conv-2	128 (7 × 7) filters with stride 2 [ReLU, L2]
Max-pooling	2 × 2 filter with stride 2
BN	-
Conv-3-1	256 (3 × 3) filters with stride 2 [ReLU, L2]
BN	-
Conv-3-2	256 (3 × 3) filters with stride 1 [ReLU, L2]
BN	-
Conv-3-3	256 (3 × 3) filters with stride 2 [ReLU, L2]
BN	-
Conv-4	512 (1 × 1) filters with stride 1 [ReLU, L2]
Max-pooling	2 × 2 filter with stride 2
Fc-5	1000 [ReLU, 0.5 dropout]
Fc-6	1000 [ReLU, 0.5 dropout]
Fc-7	3 class [(softmax classifier)]

provides the details of the designed network. First, the image size of the input layer is 100 × 100 pixels, consistent with the dataset image size. Conv-1 (11, 64) and Conv-2 (7, 128) are used to extract the primary features of the image and transfer this knowledge to the Res-Model for training. Then, {Conv3-1 (3, 128)–Conv3-2 (3, 128)} and Conv3-3 (5, 128) are used to extract high-dimensional features, and Conv-4 (1, 512) is taken to reduce the dimensions of feature information and transfer them to the fully connected layer. The batch normalization (BN) layer and regularization (L2) are used to reduce over-fitting.

In order to highlight the advantages of our model, a comparative analysis was conducted with Akram’s [15] model. As shown in

Table 3. The Differences between Akram’s CNN Model and Our Model.

Akram’s CNN Model	Our Model
Conv-1 32 (3 × 3), stride 1	Conv-1 64 (11 × 11), stride 1
Max-pooling 2 × 2, stride 2, BN	Max-pooling 2 × 2, stride 2, BN
Conv-2 64 (3 × 3), stride 1	Conv-2 128 (7 × 7), stride 2
Max-pooling 2 × 2, stride 2, BN	Max-pooling 2 × 2, stride 2, BN
Conv-3 128 (3 × 3), stride 1	Conv-3-1 256 (3 × 3), stride 2 [ReLU, L2]
Max-pooling 2 × 2, stride 2, BN	Conv-3-2 256 (3 × 3), stride 1 [ReLU, L2]-
Conv-4 256 (3 × 3), stride 1	Conv-3-3 256 (3 × 3), stride 2 [ReLU, L2]
Max-pooling 2 × 2, stride 2, BN	Conv-4 512 (1 × 1), stride 1 [ReLU, L2]
Fc-5 (512)	Max-pooling 2 × 2, stride 2
Fc-6 (2 class)	Fc-5 (1000) [ReLU, 0.5 dropout]
-	Fc-6 (1000) [ReLU, 0.5 dropout]
-	Fc-7 (3 class) [(softmax classifier)]

, 3 × 3 sized filters were employed in all conv. layers of Akram’s model, and this resulted in the inability to learn higher-level features in the initial conv. layer and subsequent conv. layers. In addition, fewer features in the full connection layer will also increase the time cost of the model, resulting in over-fitting of the classification results. In contrast, our model improves on these shortcomings. Firstly, different levels of filter banks can extract richer and more comprehensive feature information, while the use of Res-Model effectively protects the integrity of feature information. The model proposed in this article can increase the generalization capability of the network performance.

3. Experiments

3.1. Model Training

The performance of the built network is analyzed through experimental research here. The experimental study’s environment was MATLAB^® R2020b and was finished on Intel (R) Core (TM) i5-10500 CPU @3.10 GHz, NVIDIA GeForce RTX 3050 GPU, 16 GB RAM memory, with a processor of x64-based.

The experimental results were compared with AlexNet, VGG16, ResNet18 and Akram’s model [17]. AlexNet, VGG16 and ResNet18 networks are pre-trained models. In order to effectively transfer them to a new dataset for learning, the final layer of each model was redesigned considering the dataset class and then used to classify faults. Therefore, all models were trained and validated on the same dataset for the comparison of results.

In order to optimize the network, the stochastic gradient descent with momentum (SGDM) algorithm was used to minimize the loss function in the training process. Moreover, all models in deep learning rely on the minimization or maximization function. The minimized function is called the loss function. It measures the prediction performance of the prediction model by calculating the difference between the real label and the prediction label. There are many kinds of loss functions to choose from. In the above four models, the Cross Entropy Loss function was used, and the formula is as follows.

$$Loss=\frac{1}{N}{\displaystyle \sum _{i=1}^N({\displaystyle {\sum }_{x}p(x)\log q(x)})}$$

(1)

where $p(x)$ indicates the correct label of the i-th sample and $q(x)$ expresses the probability of the i-th sample.

In the experiment, taking 70% of the images of the dataset for training and the remaining 30% for validation, all models were trained with the same dataset. The input image size was 100 × 100, the mini-batch size was 64 and validation frequency and the maximum epoch were adjusted to 50. We set the learning rate as 0.001 and reduced it by a drop factor of 0.1 with every 10 epochs. The whole training process included 2750 iterations; the training process is shown in Figure 6.

3.2. Evaluation Criteria

There were many evaluation criteria for the performance of the deep learning model. Here, Precision ($Pr$), Recall ($Re$), Specificity ($Sp$), Accuracy ($Acc$) and F1-score ($F1$) are taken to quantitatively estimate the ability of the model, which was calculated through the confusion matrix. $Pr$ is the ratio of true positive samples to total predicted positive samples, $Re$ is the ratio of true positive samples to total true samples, $Sp$ is the ratio of true negative samples to predicted negative samples and $Acc$ is the ratio of correct prediction samples to total samples. $F1$ is defined as the weighted average of $Pr$ and $Re$. The principle of the mixed matrix is shown in Figure 7, and the relevant calculation methods are shown in Formulas (2)–(6) [26].

$$Pr=\frac{TP}{TP+FP}$$

(2)

$$Re=\frac{TP}{TP+FN}$$

(3)

$$Sp=\frac{TN}{TN+FP}$$

(4)

$$Acc=\frac{TP+TN}{TP+FP+TN+FN}$$

(5)

$$F1=\frac{2\times TP}{2\times TP+FP+FN}$$

(6)

3.3. Results and Discussion

Table 4. The classification performance of all methods.

Model	$\mathit{A}\mathit{c}\mathit{c} (\%)$	$\mathit{P}\mathit{r} (\%)$	$\mathit{R}\mathit{e} (\%)$	$\mathit{S}\mathit{p} (\%)$	$\mathit{F}1 (\%)$
AlexNet	93.04	91.70	91.39	96.47	91.54
VGG 16	91.25	89.49	89.89	95.66	89.67
ResNet 18	83.70	81.04	82.42	92.01	81.60
Akram’s model	94.30	92.45	92.27	96.70	92.66
Proposed model	97.42	96.72	96.68	98.76	96.70

clearly describes the classification results of all models in this dataset. It can be seen that the accuracy of AlexNet was 93.04%, that of VGG16 was 91.25%, that of ResNet18 was 83.70% and that of Akram’s model was 94.30%. The accuracy of the model proposed in this paper was 97.42%. In terms of precision, the proposed model provided the best precision result in this paper, which was 96.72%, while the precision results of AlexNet, VGG16, ResNet18 and Akram’s model were 91.70%, 89.49%, 81.04% and 92.45%, respectively. When analyzing the recall, we can see that the ResNet18 model had the worst result, which was 82.42%. Although Akram’s model provided a relatively good recall result, which was 92.27%, it was not better than the proposed model results here.

The conclusion can be drawn that the proposed model has the best classification ability compared with the other four models. For example, the specificity values of AlexNet, VGG16, ResNet18 and Akram’s models were 96.47%, 95.66%, 92.01% and 96.70%, respectively, while the result of 98.76% was obtained with the proposed model. In addition, the F1 values were 91.54% for AlexNet, 89.67% for VGG16, 81.60% for ResNet18 and 92.66% for Akram’s model, while the F1 value of the proposed method was 96.70%.

In order to better display the obtained results, the values of various indicators in Table 4 are displayed in Figure 8. In addition, the confusion matrix result of the proposed model is shown in Figure 9. There were 39 images misclassified out of 1509 validation images. It is easy to see that the proposed model is optimal in all indicators.

For each type of fault, this paper also calculated each evaluation index, and the classification performance is shown in

Table 5. The classification performance of the proposed model for three-class.

Faults	$\mathit{A}\mathit{c}\mathit{c} (\%)$	$\mathit{P}\mathit{r} (\%)$	$\mathit{R}\mathit{e} (\%)$	$\mathit{S}\mathit{p} (\%)$	$\mathit{F}1 (\%)$
hot spot	95.72	97.19	95.72	99.01	96.45
cracking	94.97	93.86	94.97	99.05	94.41
no-fault	99.35	99.10	99.35	98.21	99.22

. The classification accuracy of each type of fault reached more than 94%, and the specificity value of hot spots, cracking and no-fault types reached 99.01%, 99.05% and 98.21%, respectively. In terms of F1 index, it also reached 96.45% for hot spots, 94.41% for cracking and 99.22% for no-fault.

In order to show the classification performance of the proposed model, we compared the proposed model with four models (AlexNet, VGG16, ResNet18 and Akram’s model) on each evaluation criteria, and the improvement percentages of the model’s performance were calculated. The obtained improvement percentage results are shown in

Table 6. The improvement percentage results of the proposed model for three-class.

Model	$\mathit{A}\mathit{c}\mathit{c} (\%)$	$\mathit{P}\mathit{r} (\%)$	$\mathit{R}\mathit{e} (\%)$	$\mathit{S}\mathit{p} (\%)$	$\mathit{F}1 (\%)$
AlexNet	4.71	5.47	5.79	2.37	5.64
VGG 16	6.76	8.08	7.55	3.24	7.84
ResNet 18	16.39	19.35	17.30	7.34	18.50
Akram’s model	3.31	4.62	4.78	2.13	4.36

. In addition, the bar chart of each indicator is presented in Figure 10 for better observation and comparison.

It can be seen from Table 6 that, compared with the four models of AlexNet, VGG16, ResNet18 and Akram’s model, the improvement percentage of the proposed model in accuracy increased by 4.71%, 6.76%, 16.39% and 3.31%, respectively. The precision values improved by 5.47%, 8.08%, 19.35% and 4.62%, respectively. In addition, the recall values and specificity values were calculated as being 5.79%, 7.55%, 17.30%, 4.78% and 2.37%, 3.24%, 7.34% and 2.13% in comparison with AlexNet, VGG16, ResNet18 and Akram’s model, respectively. When the F1 values of the proposed model were analyzed, the improvement percentage of comparison AlexNet was 5.64%, that of VGG16 was 7.84%, that of ResNet18 was 18.50% and that of Akram’s model was 4.36%, respectively. Moreover, all results can be clearly observed in Figure 10. These calculation results show that the proposed model has more effective classification performance, and it is more suitable for the detection and classification tasks of PV cell faults.

In addition, in terms of time cost, we calculated the training time for five network models, as shown in

Table 7. The comparison of time cost among five models.

Model	Hardware Situation	$\mathit{A}\mathit{c}\mathit{c} (\%)$	Time Cost/ 2750 Epochs	Single Image Time (ms)
AlexNet	Intel Core i5-10500 CPU	93.04	91 min and 15 s	0.396
VGG 16	Intel Core i5-10500 CPU	91.25	340 min and 54 s	1.479
ResNet 18	Intel Core i5-10500 CPU	83.70	177 min and 40 s	0.771
Akram’s model	Intel Core i5-10500 CPU	94.30	80 min and 33 s	0.349
Proposed model	Intel Core i5-10500 CPU	97.42	30 min and 52 s	0.134

. The training time for the proposed model was about 30 min and 52 s. In other words, it took 0.134 ms to predict one image. Among the other four models, Akram’s model took the shortest time, with a time cost of about 80 min and 33 s. It can be observed that the proposed model has faster calculation, prediction speed and the highest accuracy.

4. Conclusions

In this paper, a CNN-architecture-based PV cell fault classification method is proposed, and the proposed model is trained and validated in an infrared image dataset of PV cells. In order to overcome the problem of data scarcity, an offline data augmentation method is used in the pre-processing stage to improve the fitting ability of the model. To prove the effectiveness of the proposed model, four existing models are used for comparison and quantitative analysis of the classification results. The proposed model in fault classification is 97.42%, while the accuracy of AlexNet, VGG16, ResNet18 and Akram’s models is 93.04%, 91.25%, 83.70% and 94.30%, respectively. Moreover, the proposed model is also superior to the other four models in the other four evaluation indicators. This shows that the proposed model can be well-used for the task of PV cell fault classification when infrared thermography is applied as an NDT technique.