**4. HSR Complex Background Detection Algorithm**

#### *4.1. The Complex Background That HSR Needs to Face*

HSR often needs to face a large number of external scene changes and variable terrain, environment and other influences during actual operation. These external scenes and terrain, environment, etc. can directly affect the algorithm's correct assessment of the real state of the pantograph, and thus a large number of false alarms occur. Compared with blur and dirt, which directly affect the HSC and thus affect the detection of pantographs, when these external scenes and terrain environments affect the detection of pantographs, the images captured by the HSC are still very clear and free of blobs, but their impact on pantograph detection is mainly due to the HSC imaging when these external disturbances and pantograph "overlap" together, thus causing a large number of false alarms on the pantograph state. In this study, we refer to this type of interference as the "complex background", and the common complex backgrounds are catenary support devices, the sun, bridges, tunnels, and platforms of HSR.

In this study, we propose a HSR complex background detection algorithm to achieve accurate detection of these complex scenes during the operation of HSR, so as to exclude the influence of these complex background on the pantograph state evaluation.

#### 4.1.1. Catenary Support Devices

As an extremely important part of the whole huge HSR system, the catenary support device not only plays the role of electrical insulation, but also bears a certain mechanical load. The contact network support device, as the most frequently appearing background, as shown in Figure 10 will often affect the normal detection of pantographs.

**Figure 10.** Catenary support device affects pantograph detection. (**a**) HSR-A. (**b**) HSR-B.

#### 4.1.2. Sun

As shown in Figure 11, when the sun appears in the pantograph imaging region, the strong light causes a "partial absence"-like phenomenon in the pantograph.

**Figure 11.** Sun affects pantograph detection. (**a**) HSR-A. (**b**) HSR-B.

#### 4.1.3. Bridge

Due to the complex geographical environment, when two areas are separated by rivers, only special or mixed-use bridges can be built over the rivers to provide HSR access. In more and more cities, numerous viaducts are being built to provide access to HSR. When the HSR crosses the bridge, it directly affects the detection and positioning of the pantographs. The effect of bridges on pantographs is shown in the Figure 12.

**Figure 12.** Bride affects pantograph detection. (**a**) HSR-A. (**b**) HSR-B.

#### 4.1.4. Tunnel

The presence of the tunnel greatly reduces the travel time and shortens the mileage between the two areas. Figure 13 shows the different images captured by the HSC before and after the HSR enters the tunnel. When the HSR enters the tunnel and runs stably, as shown in Figure 13c, the normal monitoring of the pantograph can still be achieved at this time because the fill light on the HSR is turned on. However, as shown in Figure 13b and Figure 13d, the dramatic light changes during the short period of time when the HSR enters

and leaves the tunnel will cause the neural network to fail to achieve accurate positioning and detection of the pantographs when entering and leaving the tunnel.

**Figure 13.** Tunnels affects pantograph detection. (**a**) Before the HSR enters the tunnel. (**b**) The moment the HSR enters the tunnel. (**c**) After the fill light is turned on, the HSR runs stably in the tunnel. (**d**) The moment the HSR exits the tunnel.

#### 4.1.5. Platform

As shown in Figure 14, when the HSR drives into the platform, the platform will partially overlap with the pantograph region, which affects YOLO's positioning and detection of the pantograph, thus causing a large number of false alarms of the pantograph status by YOLO in the platform.

**Figure 14.** Platform affects pantograph detection. (**a**) HSR-A. (**b**) HSR-B.

#### *4.2. Tunnel Detection Algorithm Based on the Overall Average Grayscale of the Image*

For such false alarms caused by drastic changes in light over a short period of time that cause YOLO to be unable to detect and locate the pantograph for a short period of time, they can be excluded by the grayscale change rule of the image. The average grayscale calculation method of the image is shown in Equation (11):

$$\overline{\mathbf{g}} = \frac{\sum\_{i=0}^{img.cols-1} \sum\_{j=0}^{img.rows-1} P(i, j)}{img.cols \ast img.rows} \tag{11}$$

where *P*(*x*, *y*) is the grayscale of the corresponding pixel point, *img*.*rows* is the height of the image and *img*.*cols* is the width of the image.

When the pantograph is running in a relatively clear and clean background, the image corresponding to each frame will cause the average grayscale of the image to fluctuate in a small range with the continuous operation of the HSR and the continuous change of the scene, but there will not be a large change in the average grayscale. Figure 15 shows the change in the average grayscale of the images taken by the HSC before and after the different cars enter and exit the tunnel.

**Figure 15.** Average grayscale variation of images of HSR-A (**top**) and HSR-B (**bottom**) when driving into different tunnels.

As can be seen from Figure 15, when the HSR is running normally outside the tunnel, the average grayscale of the image only fluctuates in a very small range, and basically remains relatively stable. When the HSR enters the tunnel, the average gray value of the captured image drops to about 5 (as shown in Figure 13b, the image is basically black) because the fill light is not yet turned on and the light inside and outside the tunnel changes drastically. As the fill light is turned on, after a short period of time to adapt to the HSR will remain in a stable state in the tunnel and continue to travel, the average gray scale of the image will remain relatively stable again (as shown in Figure 13d, the image is basically all white) and the time of the HSR in the tunnel is determined by the speed of the HSR and the length of the tunnel. When the HSR out of the tunnel, due to run from a relatively dark environment to a bright environment, the HSC overexposure phenomenon will occur. At this time the average gray scale of the HSC captured by the image will jump to close to 250 or so.

#### *4.3. Sun Detection Algorithm Based on Local Average Grayscale of Image Pantograph Region*

The influence of the sun on the HSR is full of uncertainty. We cannot accurately predict that a HSR happens to pass by at a certain time on a certain line, and the sun also happens to appear in the pantograph imaging region of the HSR at this time, and affect YOLO's assessment of the pantograph state. Moreover, not all suns are as jealous of pantograph detection as shown in Figure 11. Figure 16 shows the situation where the sun appears in some images taken by HSC, but the sun does not affect YOLO's detection of the pantograph region.

**Figure 16.** Sun did not affect YOLO detection of pantographs in HSR-A and HSR-B. (**a**) Case I. (**b**) Case II. (**c**) Case III. (**d**) Case IV. (**e**) Case V. (**f**) Case VI.

The screen of the corresponding scene in Figure 16 after the high speed rail leaves the area affected by the sun is shown in Figure 17. Furthermore, the average grayscale of the corresponding scenes in Figures 16 and 17 is shown in Figure 18.

**Figure 18.** Average grayscale comparison.

It can be found that the overall average grayscale of the image is not necessarily increased after the sun appears in the image captured by the HSC. However, when the sun affects the detection of pantographs, it will definitely cause an increase in the average grayscale of ROI. When the sun is not present the difference between the overall image and the average grayscale of the ROI is not significant, but once the sun affects the pantograph, it will definitely cause a large difference between the two. Using this unique difference, it is possible to determine whether the pantograph is detected as anomalous in the current image due to the sun. When the sun affects the pantograph detection, the average grayscale change of the overall image and ROI and the corresponding difference between the two average gray levels are shown in Figure 19.

**Figure 19.** Average grayscale variation in the corresponding areas of HSR-A (**top**) and HSR-B (**bottom**) during sun influence pantograph detection.

#### *4.4. Background Detection Algorithm for Catenary Support Devices, Bridges, and Platforms Based on Vertical Projection*

Catenary support devices, bridges, and platforms do not have an excessive effect on the average grayscale of the images captured by the HSC, so for these three common external disturbances, the choice was made to eliminate the relevant interference by using vertical projection. As shown in Figure 20a, based on the ROI positioned by YOLO V4, the left region of interest (L-ROI) and right region of interest (R-ROI) can be positioned. Firstly, the image captured by the HSC is binarized to highlight the object to be studied, and the result of binarization is shown in Figure 20b. Then the binary image is passed through the image to reduce the interference in the image by the opening operation, and the image after the opening operation is shown in Figure 20c. Finally, the vertical projection of the L-ROI, ROI, and R-ROI regions is calculated by the result of the open operation as shown in Figure 21, where the height of the white region of the vertical projection reflects the number of pixels in the white region on the corresponding horizontal coordinates in the binary image.

**Figure 20.** Image binarization and opening operations. (**a**) L-ROI, ROI and R-ROI. (**b**) Binary image. (**c**) Binary image after opening operation.

**Figure 21.** Binary image of different regions and the corresponding vertical projections after the opening operation. (**a**) L-ROI. (**b**) ROI. (**c**) R-ROI.

As shown in Figure 22, the percentage of white areas in the vertical projections of L-ROI and R-ROI is low when the HSR is operating normally without external disturbance, while there is a large percentage of white areas in the vertical projections corresponding to ROI.

**Figure 22.** Change in the percentage of white areas in the vertical projection of different areas of HSR-A (**top**) and HSR-B (**bottom**) when the HSR is operated without external disturbances.

The impact of the catenary support device on the pantograph detection is much smaller compared to other complex backgrounds, but the percentage of white areas in the vertical projection still reflects the changes brought about by this scenario very accurately. The changes in the percentage of white areas in the vertical projection after different areas in the L-ROI, ROI and R-ROI are affected by the catenary support devices during the operation of the HSR are shown in Figure 23.

**Figure 23.** Changes in the percentage of white areas in the vertical projections of different areas of HSR-A (**top**) and HSR-B (**bottom**) during HSR operation after being affected by the catenary support devices.

The effect of bridges on the percentage of white areas in the vertical projection of different regions during HSR operation is shown in Figure 24. Since the HSC angles of HSR-A and HSR-B are different, the bridges do not have the same effect on the percentage of white in the vertical projection areas of L-ROI and R-ROI, but both cause at least one of the L-ROI or R-ROI to have a huge change in the percentage of white area in the vertical projection.

**Figure 24.** Changes in the percentage of white areas in the vertical projections of different areas of HSR-A (**top**) and HSR-B (**bottom**) during HSR operation after being influenced by the bridge.

The effect of the platform on the percentage of white areas in the vertical projection of the different areas is shown in Figure 25. Furthermore, due to the HSC angle, the impact of

the platform on HSR-A and HSR-B is different, but both have an impact on at least one of the R-ROI or L-ROI.

**Figure 25.** Changes in the percentage of white areas in the vertical projections of different areas of HSR-A (**top**) and HSR-B (**bottom**) during HSR operation after being influenced by the platform.

From Figures 22–25, it can be seen that the percentage of white area in the projection corresponding to ROI does not change much when subjected to complex background interference, while the changes of L-ROI and R-ROI are very obvious after subjected to complex background interference, so this paper mainly detects the presence of complex background interference by the projection of L-ROI and R-ROI areas.

#### *4.5. Overall Process of HSR Complex Background Detection Algorithm*

The overall process of the complex background detection algorithm is shown in Figure 26. For a pantograph image captured by a HSC, when it cannot be detected or is detected as abnormal, the complex background detection algorithm is needed to assess whether the current detection result has the possibility of being affected by the complex background.

The specific process is as follows: First, the change of the average grayscale of the current image as a whole and the average grayscale of the previous frame as a whole is used to evaluate whether the detection result may be affected by the drastic change of light before and after the HSR enters and leaves the tunnel. If not, the relationship between the overall average grayscale of the image and the average grayscale of the ROI is used to assess whether the sun may have intruded into the pantograph region and thus influenced the pantograph detection. If the influence of the sun can still be excluded, the detection of the catenary support devices, platforms, and bridges is achieved by vertical projection to finally determine whether the pantograph detection results are influenced by the complex background at this time.

If the influence of complex background on the detection result is excluded by HSR complex background detection algorithm, then there are still two possibilities for the pantograph not to be detected or detected as abnormal: (1) although the current image is not disturbed by complex background, it may be disturbed by other interference which leads to misjudgment of the pantograph, (2) the pantograph does appear abnormal. In this case, the overall algorithm proposed in Section 5.1 of this study is combined to achieve accurate detection of the real situation of pantographs.

**Figure 26.** HSR complex background detection algorithm process flow chart.
