Research on Digital Meter Reading Method of Inspection Robot Based on Deep Learning

Abstract

Aiming at solving the issue of blurred images and difficult recognition of digital meters encountered by inspection robots in the inspection process, this paper proposes a deep-learning-based method for blurred image restoration and LED digital identification. Firstly, fast Fourier transform (FFT) is used to perform blur detection on the acquired images. Then, the blurred images are recovered using spatial-attention-improved adversarial neural networks. Finally, the digital meter region is extracted using the polygon-YOLOv5 model and corrected via perspective transformation. The digits in the image are extracted using the YOLOv5s model, and then recognized by the CRNN for digit recognition. It is experimentally verified that the improved adversarial neural network in this paper achieves 26.562 in the PSNR metric and 0.861 in the SSIM metric. The missing rate of the digital meter reading method proposed in the paper is only 1% and the accuracy rate is 98%. The method proposed in this paper effectively overcomes the image blurring problem caused by the detection robot during the detection process. This method solves the problems of inaccurate positioning and low digital recognition accuracy of LED digital meters in complex and changeable environments, and provides a new method for reading digital meters.

1. Introduction

With the continuous development of society, people’s demand for electricity is also increasing. In order to reduce the loss of a power outage caused by power equipment failure and ensure the safe operation of power equipment, it is necessary to conduct regular inspections of power equipment. Equipment is usually inspected via manual inspection, where staff regularly check the equipment and record the results according to the designated route. This method has many issues, such as prolonged inspection time, low inspection efficiency and high omission rate.

In recent years, with the continuous breakthrough of technology, more and more machine learning technologies have been applied to various industrial fields. For example, the physics-informed neural network (PINN for short) [1] is a type of neural network used to solve supervised learning tasks. Compared with pure data-driven neural network learning, PINN imposes physical information constraints in the training process, and so it can learn a model with more generalization ability and fewer data samples. Misyris et al. [2] applied the PINN in the power system to achieve an accurate prediction of rotor angle and frequency based on small sample conditions. Ma et al. [3] proposed a resonance-informed deep learning (RIDL) strategy for fast and accurate prediction of the optical response of ultrahigh Q-factor resonances. By incorporating resonance information into deep learning algorithms, the RIDL strategy achieves highly accurate predictions of reflection spectra and photonic band structures while using relatively small training datasets. Additionally, contributions from machine learning techniques can be seen in object detection and other areas. For instance, You Only Look Once (YOLO for short) [4] is an end-to-end target detection algorithm based on neural networks. YOLO has good real-time and accurate detection, making it popular in the detection field. Yao et al. [5] realized the real-time detection of tiny defects in kiwi fruit based on an improved YOLOv5 [6] algorithm. Xu et al. [7] realized the automatic grading of apples using the improved YOLOv5 algorithm, which solved the problems of low grading accuracy and slow grading speed in the apple grading process. Furthermore, Ji et al. [8] used image enhancement based on multi-scale pyramid fusion and the MobileCenterNet model to realize the real-time detection of underwater crabs. Xiang et al. [9] realized the text recognition task of the pill box based on DBNet and the CRNN [10].

In the field of power industry inspection, machine learning technology has also been widely used. For example, Tian et al. [11] located the position of the pointer by searching the minimum bounding rectangle (MBR) of the pointer region, and then used the connection between the center of the pointer to the center of the dial as the pointer and calculated the indication value corresponding to this pointer according to the angle method. Zhang et al. [12] proposed a meter recognition and reading method based on the YOLOv3 [13]. This method used the trained YOLOv3 model to detect and identify meter types. The dial of the pointer meter is detected by the Hough circle detection algorithm and the scale and pointer are extracted using the image processing method. This method calculated the angle between the 0-scale line and the pointer to convert the reading of the pointer meter. For digital instruments, this method obtained digital areas via image morphology and recognized them using a trained support vector machine (SVM). Duan et al. [14] used an SVM to identify the extracted digits. Hou et al. [15] used Mask R-CNN [16] to locate the dial position of a meter. Next, all of the digit scale regions were identified using RCNN [17] and the pointer was extracted using the region growing method. The reading was then based on the position of the pointer and the two closest scale marks found by searching through the circular scale. Deng et al. [18] used a cascade approach toward meter reading. Firstly, the trained YOLOv5 [19] was used to obtain dashboards. Then, the improved Deeplabv3 [20] was used to segment pointers and numbers to realize the pointer meter reading. Fabio et al. [21] used YOLOv5s to locate and identify the digits of the meter. The effectiveness of the proposed automatic digital position identification method for digital meters was verified via practical experimental cases. Although the above-mentioned methods can obtain gauge representation, none of them take into account image blurring. Therefore, the accuracy is low when recognizing blurred images.

Most of the existing inspection robots adopt the mode of fixed-point inspection. The robot moves to a fixed position and stays there to complete image acquisition and detection, and then goes to the next designated inspection point. Although this approach can ensure the stability of detection, it greatly increased the inspection time. In order to reduce the inspection time and improve the inspection efficiency, the inspection strategy of “walk-and-inspect” was proposed in this paper, which means that the robot inspects the equipment during its movement, and the robot does not need to stop during the image acquisition and inspection. The inspection method can save a lot of time, but the image acquisition during the robot’s movement will cause image motion blur.

In order to ensure that the inspection robot can complete the inspection task better, it is necessary to detect and restore the blurred image. Image blur detection methods are usually divided into spatial-domain-based image blur detection and frequency-domain-based image blur detection. Methods such as the Tenengrad gradient method [22], Laplacian gradient method, energy gradient method and variance method [23] are often used for image blur detection. The advantages of these types of method are simplicity and quick speed, but the disadvantage is that the threshold value is difficult to determine due to the susceptibility to illumination. In summary, the image blur detection method based on fast Fourier transform in the frequency domain has better stability. In order to guarantee the quality of images, it is necessary to recover images with motion blur. Traditional image motion blur recovery algorithms such as the Wiener filtering method [24], edge function method [25] and PSF method [26] have complicated blur kernel estimation processes and the deblurring effect depends largely on the priori assumptions of the image; so, it is difficult to be extended to other types of blurs. Compared to the traditional image motion blur recovery algorithms, the deep-learning-based image motion blur recovery methods achieve motion blur recovery in different scenes by sensing and analyzing image blur information. Deep-learning-based image blur recovery methods had better generalization and robustness. In particular, the deep learning method based on the adversarial neural network cannot only preserve the texture features of the original image, but also has a good effect on the contour recovery of the image. For the image blurring issue caused during motion detection, an image blur recovery method based on an improved deblurring algorithm is suggested. The method can overcome the interference of ambient light changes and achieve the highly robust recovery of motion-blurred images.

In addition, for the recognition of LED digital meter readings commonly used in power inspection tasks, traditional detection algorithms have problems such as poor robustness and a high false detection rate. In view of the above situation, this paper adopts the “coarse to fine” detection strategy combined with deep learning technology. Firstly, the Polygon-YOLOv5 [27] model was used to detect the LED digital area and correct the LED digital area with perspective transformation. Then, the YOLOv5 model was used for the precise positioning of digits line by line. Finally, the CRNN model was used to recognize the numbers and complete the recognition task of the LED digital meter reading. The method in this paper solves the problems of poor robustness and high false detection rate of LED digital meter reading recognition caused by changes in lighting and camera shooting angles. The rest of the paper is arranged as follows: Section 2 introduces the image blur detection method based on fast Fourier transform and the image blur restoration method based on an improved adversarial neural network; Section 3 describes the digital meter detection based on the coarse-to-fine strategy and the reading recognition method based on the CRNN; Section 4 demonstrates the accuracy and effectiveness of the proposed method and, finally, Section 5 summarizes the work of the paper and the prospects for future research issues.

2. Image Blur Detection and Restoration

The quality of the image will directly affect the detection results of the image; so, it is critical to improve the quality of the image. Image quality loss mainly comes from image blurring and noise interference. Noise interference refers to the appearance of randomly distributed points or lines in the image that are unrelated to the image. Image blur is mainly classified as out-of-focus blur and motion blur. The blur brought about by objects in the scene outside the imaging depth of the field is called out-of-focus blur. In the imaging process, the image blur brought about by camera movement or scene change caused by the overlap of different spatial location information is called motion blur. In order to improve the accuracy of the detection work, an FFT-based image blur detection method is used to achieve the quality assessment of images. Meanwhile, in this paper, a method based on a spatial-attention-improved adversarial neural network was used for blurred image recovery.

2.1. Image Blur Detection Based on FFT

Discrete Fourier transform (DFT) is a discrete form of Fourier transform in the time and frequency domains, which converts a sample of the time domain signal into a sample of the discrete time Fourier transform (DTFT) frequency domain. In realistic applications, FFT is often used to efficiently compute the discrete Fourier transform. For an image of size N × N, the two-dimensional discrete Fourier (DFT) formulation is shown in Equations (1) and (2).

$$F(k,l)={\displaystyle \sum _{i=0}^{N-1}{\displaystyle \sum _{j=0}^{N-1}f(i,j)e^{-i2\pi (\frac{ki}{N}+\frac{lj}{N})}}}$$

(1)

$$e^{ix}=\cos x+i\sin x$$

(2)

where f (i, j) represents the image in the spatial domain, F (k, l) represents the image in the frequency domain, and $e^{ix}$ is the basis function corresponding to each point in the Fourier space.

In the frequency domain, the high-frequency part represents the texture features of the image, while the low-frequency part represents the contour information of the image. Therefore, after the FFT processing of the image, the frequency domain amplitude of the image can be calculated via spectrum analysis, and the image can be judged as to whether it is blurred, as shown in Figure 1.

In this paper, the average value of the spectral amplitude of an image was calculated to determine whether the image is blurred or not. When the average value of the spectral amplitude of an image is lower than a set threshold, it is determined to be a blurred image; otherwise, it is a normal image. The image blur detection process is shown in Figure 2.

2.2. Deep-Learning-Based Image Motion Blur Restoration

For the image blurring issue, the DeblurGANv2 [28] model proposed by Kupyn [29] was used in this paper. DeblugGANv2 is a fully convolutional model, in which the FPN (feature pyramid network) is used as the core building block of the network to extract multi-dimensional feature information. The feature information contains different image information codes, and through which different levels of image semantic features can be obtained. These information features are reconstructed by an upsampling layer and the information features are interrelated. In addition, the upsampling layer and the convolution layer are used to restore clear images and remove smudges. The pipeline architecture of DebulugGAN-v2 is shown in Figure 3.

The simple merging method in the DeburGAN-v2 structure results in a loss of image features. In order to improve the effect of semantic segmentation, the spatial attention module is added to this pain. The spatial attention module obtains the details and edge information of the image from the low-order features and the clear and effective spatial regions from the high-order features. The weights of the spatial attention module are used to adjust the attention so that the model can focus on the regions that deserve more attention, which improves the performance of the model.

In particular, the instance specification layer is replaced by the adaptive instance normalization (AdaIN) layer [30], and the ReLU activation function is replaced by the SiLU (s-type weighted liner unit) activation function [31]. The AdaIN layer can compute the linear coefficients after style transfer from a small number of images, so as to make the reconstructed image more accurately reflect the real image. In the feature layer, the AdaIN layer achieves style transfer with less computational overhead by changing the data distribution of the features. The function of AdaIN is shown in Equation (3). The SiLU activation function has no upper bound, no lower bound, no smoothing and no non-monotonicity, which is better than the ReLU activation function in the depth model. The improved network structure is shown in Figure 4.

$$\mathrm{AdaIN}(x,y)=\sigma (y)\left(\frac{x-\mu (x)}{\sigma (x)}\right)+\mu (y)$$

(3)

where x represents content input and y represents style input, µ(x) and σ(x), respectively, represent the mean and standard deviation of the batch size and spatial size calculated independently by the feature channel and σ(y) scales the normalized content input and shifts it by µ(y).

3. Digital Meter Detection and Identification

A “coarse-to-fine” two-stage inspection method is proposed in this paper for the inspection of digital meters. Firstly, the meter was detected according to PolygonYOLOv5 and corrected using perspective transformation. Then, the numerical region was accurately detected using YOLOv5s. Finally, the numbers were identified by the CRNN. The strategy in this thesis reduces the probability of meter misses and improves the accuracy of the readings, as shown in Figure 5 and Figure 6.

3.1. The Model of Polygon-YOLOv5

Adjusting the important parameters of the deep learning model according to different task requirements can make the model achieve good results under different task requirements. The YOLO family of models is leading among various target detection models [32]. YOLOv5, as a member of the YOLO family [33], has significant advantages in terms of processing time optimization for multi-layer networks. In view of the excellent performance of YOLOv5, YOLOv5 was used in this paper for the digital area recognition of meters.

Different from the rectangular prediction box of the YOLOv5, Polygon-YOLOv5 can reach the regional detection of irregular targets. As shown in Figure 7, the rectangle prediction box of YOLOv5 is the red dotted line box, which includes coordinates of the box’s upper left corner, width and height of the box. The irregular prediction box of polygon-YOLOv5 is represented by Corner1, Corner2, Corner3 and Corner4. Additionally, the coordinates of the irregular box meet y₃ ≥ y₁&y₃ ≥ y₂&y₄ ≥ y₁&y₄ ≥ y₂&x₁ ≤ x₂&x₄ ≤ x₃ conditions. The corners and relative positions of the corner points and target grid center of the irregular box are predicted through polygon-YOLOv5. Additionally, variant polygon object detection is performed by setting the different corners.

3.2. Perspective Transformation

The detection target area and the four corners of the detection target area are obtained from the polygon-YOLOv5 model. The homography matrix can be computed through the four corner points and the width and height of the LED digital meter. The LED digital meter area is corrected according to the perspective transformation. The perspective transformation formula is demonstrated below.

$$\left[x^{\prime },y^{\prime },w^{\prime }\right]=[u,v,w]\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\\ a_{31}& a_{32}& a_{33}\end{array}\right]=[u,v,w]\left[\begin{array}{cc}{T}_1& T_2\\ T_3& a_{33}\end{array}\right]$$

(4)

where parameter $w=1$, ($u,v$) is the raw image coordinates, ($\frac{x^{\prime }}{w^{\prime }},\frac{y^{\prime }}{w^{\prime }}$) is the image coordinates obtained after perspective transformation, $T_1=\left[\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right]$ is the image linear transformation matrix, $T_2={\left[\begin{array}{cc}{a}_{13}& a_{23}\end{array}\right]}^{\mathrm{T}}$ is the image homography matrix, $T_3=\left[\begin{array}{cc}{a}_{31}& a_{32}\end{array}\right]$ is the image shift matrix and parameter $a_{33}=1$. Figure 8 shows the image after correction.

3.3. Digit LED Recognition of Meter by CRNN

The CRNN is a convolutional recurrent neural network structure that is used primarily for the end-to-end recognition of text sequences of uncertain length. Instead of cutting an individual text, it transforms text recognition into a sequence learning issue that relies on time sequences and image-based sequence recognition. The CRNN structure consists of three main parts, a convolutional layer, a recursive layer and a transcription layer. The CRNN structure is shown in Figure 9.

4. Experiments and Discussion

4.1. Motion Blur Restoration Experiment

In this paper, a dataset of 500 pairs of blurred images was generated from the collected normal images. The generated blurred image datasets were used to train the DeblurGAN algorithm and the improved DeblurGAN algorithm in this paper. The peak signal-to-noise ratio (PSNR) and structural similarity ratio (SSIM) of the images were used as evaluation metrics for image motion blur restoration. The larger the PSNR and SSIM values of an image, the smaller the distortion of the image.

Table 1. PSNR and SSIM of DeblurGAN and the improved DeblurGAN in this paper.

	DeblurGAN	Improved DeblurGAN
PSNR	25.534	26.562
SSIM	0.770	0.861

shows the comparison of the improved model and the original model metrics in this paper. It can be clearly seen from Table 1 that the original network structure has a PSNR index of 25.534 and SSIM index of 0.770 for blurred image recovery, while the improved model in this paper has a PSNR index of 26.562 and SSIM index of 0.861. Obviously, the improved network outperforms the original network in blurred image recovery. Meanwhile, Figure 10 shows the effect of the original network and the improved network on the recovery of the blurred digital image. It is obvious from Figure 10c that the improved network is significantly more effective than the original network in terms of blurred image recovery.

4.2. Reading Experiment of LED Digit Meter

Experimentally, 300 image data were collected and expanded to 900 via data enhancement for training the YOLOv5s model and Polygon-YOLOv5 model. In order to ensure the training effect of the CRNN model, this paper used the TRG (text recognition data generator, TRG) algorithm to randomly generate analog numbers. The algorithm generates digital images in a specific context and adjusts the tilt angle, blur and digit size attributes of the generated digits based on the actual digital image. The digital image generated using the TRG algorithm can be closer to the digital image in the real scene. The digital image generated using the simulation is shown in Figure 11.

To verify the effectiveness of the method in this paper, 500 images with LED digital meters collected in different time periods were tested. The experiments included the detection and reading recognition of LED digital meters.

Table 2. The detection results of the digital meter with different detection algorithms.

	Missing Rate of Meter Region	Error Rate of Meter Region
SVM	4.5%	2.5%
YOLOv5s	3%	1.5%
Our method	1%	0%

shows the miss detection rate and error rate of our method with YOLOv5s for LED digital meter detection. Figure 12 shows the effect of digital meter detection and recognition in different time periods. Additionally, it can be seen from Figure 12 that the experimental method can read the meter accurately in different time periods.

To verify the effectiveness of the meter detection recognition, this paper compared SVM and the YOLOv5s and Polygon-YOLOv5 methods for digital meter detection. From Table 2, it is clear that the Polygon-YOLOv5 meter miss detection rate is 1% and the false recognition rate is 0%. Its performance is much better than SVM and YOLOv5s. Meanwhile, to verify the effect of different positioning methods on the final digital meter readings, different positioning methods were chosen for experimental comparison in the paper. It can be seen from

Table 3. The detection results of the digital meter reading with different detection algorithms.

	Accuracy of Reading
SVM	79%
YOLOv5s	85%
YOLOv5s+ minimum enclosing rectangle	89%
Our method	98%

that the reading accuracy using the SVM method is only 79%; the reading accuracy using YOLOv5s is 85%, and the reading accuracy using the YOLOv5s+ minimum circumscribed rectangle method is 89%. The positioning method proposed in this paper has a reading accuracy of 98%. It is not difficult to see that the positioning method adopted in this paper performs far better than the other positioning methods in terms of reading accuracy.

In the actual inspection environment, the recognition accuracy of the CRNN model for numbers is easily affected by changes in illumination. Figure 13 shows the original image and its BGR three-channel image under three different lighting conditions of the digital image. In order to obtain more effective digital features as much as possible and ensure the accuracy of the readings, this paper preprocessed the digital images. It can be seen from Figure 13b that when the digital image is overexposed, the gray level of the red channel image is too high, and the gray levels of the other two channels are normal. It can be seen from Figure 13a,c that when the digital image is normal or weakly exposed, the gray value of the green channel image is almost negligible, and the gray value of the other two channel images is slightly lower. Based on the above situation, this paper calculated the standard deviation of the gray value of the red channel of the image. If the standard deviation was greater than the set threshold, the blue and green channel images were superimposed. Otherwise, the red channel image was enhanced and superimposed with the blue channel. As shown in Figure 14, this is the effect after digital image preprocessing.

In this paper, 500 digital images were randomly selected to test the reading accuracy of the CRNN model before and after preprocessing.

Table 4. The reading accuracy of the CRNN model before and after preprocessing.

	Accuracy of Reading
Without preprocessing	86.4%
Preprocessing	98.8%

shows the reading accuracy of the CRNN model before and after preprocessing. It can be seen from Table 4 that the reading accuracy of the CRNN after preprocessing reaches 98.8%, which is significantly improved compared with that without preprocessing. This paper analyzed the false reading rate of 1.2%. The main reason for the false readings of the CRNN model is that part of the digital area is missing due to light reflection. In future research work, we will consider adding devices such as polarizers to the hardware to eliminate the interference of some external light.

5. Conclusions

For the issue of blurred images and difficult LED digital meter reading recognition caused by the moving inspection process of the inspection robot, this paper proposes a deep-learning-based method for blurred image repair and LED digital meter reading recognition. First, the image was processed via FFT and the average value of the image amplitude was counted to determine whether the image was blurred or not. Then, the blurred image was restored using an improved adversarial neural network. Finally, the LED digital meter in the image was represented by the detection strategy “coarse-to-fine” proposed in this paper. The experiments proved that the method in this paper cannot only effectively overcome the image blur caused by the motion of the detection robot, but can also achieve accurate reading recognition of the LED digital meter in complex environments. The method in this paper provides an efficient and robust detection idea with practical value in the field of detection robots. In future work, we will continue to study the application field of inspection robots based on machine vision, so that the inspection work will develop in the direction of unmanned intelligence.