A Seam Tracking Method Based on an Image Segmentation Deep Convolutional Neural Network

Abstract

Vision-based welding seam tracking is an important and unique branch of welding automation. Active vision seam tracking systems achieve accurate feature extraction by using an auxiliary light source, but this will introduce extra costs and the real-time performance will be affected. In contrast, passive vision systems achieve better real-time performance and their structure is relatively simple. This paper proposes a passive vision welding seam tracking system in Plasma Arc Welding (PAW) based on semantic segmentation. The BiseNetV2 network is adopted in this paper and online hard example mining (OHEM) is used to improve the segmentation effect. This network structure is a lightweight structure allowing effective image feature extraction. According to the segmentation results, the offset between the welding seam and the welding torch can be calculated. The results of the experiments show that the proposed method can achieve 57 FPS and the average error of the offset calculation is within 0.07 mm, meaning it can be used for real-time seam tracking.

1. Introduction

In traditional off-line welding modes, the trajectory of the welding torch needs to be set in advance. The acquisition of the motion trajectory mainly depends on advance measurements to calculate the motion trajectory and pass the information to the robot. However, these methods cannot adjust the position of the welding torch according to the situation during the welding process, causing some precision problems. The development of automation technology has had a huge impact on the field of robotic welding, and various forms of welding involving automation solutions based on different physical features are constantly being proposed. Among them, vision-based welding automation has unique advantages in terms of feature extraction [1]. Compared with traditional welding solutions, vision-based automated welding solutions significantly improve the work efficiency and welding quality. The traditional welding quality improvement strategy mainly relies on the use of additional equipment to improve the welding quality, but it still requires a lot of manual work. On the contrary, vision-based automated welding robots can select welding paths and welding parameters for different welding conditions without relying on experience to gradually set the relevant parameters [2]. Vision-based welding automation approaches can easily and intuitively obtain image features of the welding arc and seam, which can greatly facilitate the feature extraction required for the subsequent welding seam tracking.

When designing vision-based seam tracking systems, there are also certain problems to be solved. To obtain high-precision tracking results, it is necessary to keep the camera’s field of view at a position with a small distance from the molten pool. However, the closer the position to the molten pool, the stronger the interference of the arc light, which makes it more difficult to detect the welding seam [3]. In addition, the features of the welding seam are inconspicuous because of the tight butt join of the splice plate and the small groove. Effective solutions to these problems are required when designing vision-based weld tracking systems.

According to whether the welding seam tracking system requires auxiliary light sources, it can be classed as an active vision welding seam tracking system or passive vision welding seam tracking system. In active vision welding seam tracking, the structured laser light is generally used as an auxiliary light source, and the features of the structured light are extracted to complete the welding seam tracking. Kawakara et al. built a welding seam tracking system with a laser line-structured light, and used the image sequence information to eliminate the noise in the weld image [4]. Kim et al. adopted a text analysis method to process the laser stripes in an image to improve the robustness of the system, which reduced the influence of the strong arc light and metal splash [5]. Jawad et al. used the improved Otus algorithm to segment the laser stripes and complete the fitting of the laser lines [6]. Yang et al. proposed a seam tracking system based on an adaptive Hough transform to achieve the real-time extraction of laser stripe features [7]. Zou et al. applied a continuous convolution operator tracking algorithm (CCOT) in their system to achieve real-time seam tracking, and a Histogram of Oriented Gradient (HOG) was used to extract the feature [8]. The rapid development of convolutional neural network-related technologies has brought a new direction to seam tracking. Xiao et al. built a seam tracking system based on the Faster R-CNN, which can automatically identify the seam type and extract the seam edge [9]. Zou et al. achieved welding seam detection based on an Single Shot MultiBox Detector (SSD) detector with a multi-feature fusion network [10]. Zhao et al. proposed an image segmentation method based on deep learning to extract welding seam features, which can achieve highly robust welding seam tracking [11]. To sum up, active vision systems mainly focus on improving the effect of the laser line feature extraction to obtain more precise tracking results.

Since the active vision relies on the illumination of the auxiliary light source, the seam features are more obvious. However, active-vision-based systems require additional structured light equipment. In addition, when building such systems, it is necessary to ensure the coordination of the welding torch, laser, and camera. The seam position predicted by active vision system is relatively far from the torch position because the features of the laser line are affected by the melt pool if the distance between the welding torch and the laser line is close. Therefore, the real-time performance of the active vision system will be effected. Passive vision-based systems have some advantages in actual production because they avoid the additional cost of an auxiliary light source and the system structure is relatively simple. Ge et al. performed grayscale feature extraction on the molten pool area and determined the welding offset by extracting the center and boundary positions of the molten pool [12]. Wei et al. detected the edge with a Sobel operator and Canny operator to determine the seam position [13]. Xu et al. used an improved Canny edge detection method to detect the edges of the seam and arc using two region of interest (ROI) and calculated the offset of the welding torch from the seam [14]. Shao et al. designed an image processing algorithm based on the particle filter method to track the seam [15]. Chen et al. achieved welding seam tracking based on the Mask-RCNN to segment the molten pool area, and a Hough line transformation was used to fit the seam line [16]. The current research on passive vision welding seam tracking involves performing line fitting on the seam part directly, then performing image segmentation on the weld pool or arc part. The image processing requires different processing methods for the molten pool and the seam. None of these methods adapt well to changing conditions when the torch deviates too far from the seam because of the ROI processing.

Aiming at the problems existing in the current passive vision welding seam tracking research, a passive vision welding seam tracking system based on a semantic segmentation neural network is proposed in this paper, which can be used for end-to-end image segmentation. The aim of our work is to detect the position of the welding seam in real time and guide the path of the welding torch. The proposed method segments images based on deep learning, and can directly distinguish the welding arc and seam from the images. After the image segmentation, the connected component analysis is used to process the mis-segmented part in the image. Then, the positions of the welding seam and the welding torch are calculated according to the semantic image. Finally, the offset between the welding torch and the seam is calculated to accomplish seam tracking and a filter method is proposed to improve the precision.

2. Passive Vision Weld Seam Tracking System

The passive welding seam tracking vision system used in this paper consists of an industrial camera and a light reduction film. The industrial robot model used here is a Yaskawa robot, the welding controller is a Fronius3000 (Fronius, Austria), and the industrial camera is a Basler acA640-750uc (Basler, Allensburg, Germany). During the welding process, the camera is fixed on the welding torch, the positions of the welding torch and the camera are relatively static, and the field of view of the industrial camera is 40 mm × 20 mm. The angle of the camera is about 30 degrees from the horizontal plane and the imaging distance is 450 mm. A light reduction film is installed in the front of the camera to reduce the influence of the arc light during the welding process, so that the camera can capture clear arc and seam images for subsequent feature extraction. The passive vision seam tracking system is shown in Figure 1, and the state of the system during welding is shown in Figure 2.

In this study, the welding of splice plates under the PAW welding process is the main focus. Compared with other welding processes, the arc light of the PAW is relatively stable, meaning it can provide a stable lighting source for passive welding seam tracking. Moreover, the PAW can avoid the influence of metal spatter on the semantic segmentation, which is convenient for collecting more stable welding images.

3. Algorithm

In this paper, semantic segmentation is used to obtain the positions of the arc and seam. The main idea is to use multi-category semantic segmentation to determine the positions of the welding seam and the arc according to the image features of different parts in welding scene, and to calculate the offset between them. The overall flow chart of the welding seam tracking is shown in Figure 3.

As an important technology in the field of computer vision technology, semantic segmentation has made great progress in recent years. The traditional semantic segmentation approach mainly assigns semantics according to the gray value of each pixel, including via gray-threshold-based algorithms, region-growing algorithms, and superpixel segmentation. With the rapid development of neural network technology, more researchers have proposed semantic segmentation algorithms based on deep learning. Most semantic segmentation is based on two-network backbone structures: a dilation backbone network and encoder–decoder backbone network. Semantic segmentation based on these two network structures can achieve excellent segmentation results, but cannot meet the real-time segmentation requirements in specific situations. Some subsequent real-time semantic segmentation networks such as SegNet can achieve real-time segmentation, but the segmentation accuracy is decreased. To achieve a win–win result during real-time segmentation and to improve the segmentation accuracy, Yu et al. built a lightweight real-time semantic segmentation network named BiseNetV2 based on their previous work [17]. This paper adopts this network structure to build a passive vision seam tracking system.

3.1. Network Structure

The structure of BiseNetV2 is mainly composed of three parts: the detailed branch, semantic branch, and aggregation layer. The first two parts belong to the backbone part, and the aggregation layer is used to fuse the features extracted by these two branches. The specific network structure is shown in Figure 4.

The function of the detailed branch is mainly to extract the detailed information from the image. The representation of the detailed information is diverse, so the representation in the feature map requires a high channel capacity. Therefore, the network structure of the detailed branch needs to be designed with wide channel dimensions and shallow layers to achieve the extraction of the detailed information. Generally, the network architecture of VGGNet can be directly adopted. The specific structure is three convolution layers combined with batch normalization and activation functions, the size of the convolution kernel is 3, the stride is 2, and the final feature map size output by this branch is 1/8 of the original image.

Corresponding to the detailed branch, the role of the semantics branch is mainly to extract high-level semantic features from images. This branch does not need to care about the detailed features in the image, so its channel can be narrow. A fast downsampling strategy is adopted in this part of the structure to enlarge the receptive field quickly for the high-level semantics requirement. Global average pooling is used to embed the global contextual response. Since this branch channel is narrow, it is a lightweight structure and can be implemented by any efficient model. In the actual structure implementation process, the stem block structure [18] is adopted in the first stage of the semantics branch, which uses two different downsampling methods to shrink the feature representation. The context embedding block structure inspired from [19] is adopted at the end of this branch, which adopts the global average pooling and residual connection processes to insert the global context information. The intermediate network structure uses a gather-and-expansion layer, and its main structure consists of three parts: (i) a size 3 convolution kernel to aggregate features and expand to the high-dimensional space; (ii) a 3 × 3 depthwise convolution layer performed independently over each individual output channel of the expansion layer; (iii) a size 1 convolution kernel to project the output of the depthwise convolution into the low channel capacity space.

After the feature extraction of the detailed branch and the semantic branch, we need to merge the features with different widths and depths from these two branches. Therefore, an aggregation layer needs to be connected behind the two branches to merge the feature representations with different resolutions. In order to better merge the complementary information for the two branches, BiseNetV2 uses a bilateral guided aggregation layer for aggregation. The key of the specific implementation is to use the global context information for the semantic branch to optimize the feature distribution of the detailed branch. In this way, the multi-scale information can be intrinsically encoded, which is beneficial for obtaining feature distributions at different resolutions. Moreover, this method increases the efficiency of the feature communication between branches.

3.2. Online Hard Example Mining

During the network training, the contributions of different samples to the loss function will be different. The parts that contribute more to the loss function can be divided into two categories: (1) hard positives, which are positive samples that are easily classified as negative samples; (2) hard negatives, which are negative samples that are easily classified as positive samples. In order to overcome the impact of the unbalanced samples on the network training, we need to process the hard samples to balance the positive and negative samples.

Abhinav et al. [20] proposed a hard sample training strategy called online hard example mining (OHEM) for object detection networks. The main idea is to take the ROI to select the hard samples instead of sampling the dataset. When performing the backpropagation, the parts of the ROI samples with the maximum loss are selected to update the weights. The ROIs with similar positions have similar losses. In order to avoid the influence of the multiple selection of similar ROIs on the training results during backward propagation, it is necessary to perform non-maximum suppression on the ROIs of difficult samples. In the forward pass, the network calculates the feature maps of all ROIs and obtains the losses of all ROIs. Then, a hard ROI module is used to filter the loss and train these hard samples separately. The loss of the OHEM is as follows:

$$L=-\frac{1}{N}{{\displaystyle \sum }}_{k=1}^K{w}_k{{\displaystyle \sum }}_{n=1}^N[f_{kn}\ast \log \left(p_{kn}\right)+\left(1-f_{kn}\right)\ast \log \left(1-p_{kn}\right)]$$

Among them, $w_k$ is the weight of each category, $K$ is the number of categories, $p_{kn}$ is the gray value of the nth pixel of the $k$ category in gt, $f_{kn}$ is the predicted value of the corresponding pixel, and $N$ represents the difficult pixel selected by the OHEM.

3.3. Image Processing and Offset Calculation

After image segmentation, the semantic image needs to be post-processed to reduce the calculation error of the offset. During the training process of the semantic segmentation network, the convergence value of its parameters cannot fit all data samples, which will result in unsatisfactory segmentation of some images. Some segmented images are shown in Figure 5. It can be seen that since the arc part has obvious features and occupies a large proportion of the number of pixels, the segmentation of the arc part is relatively stable. In contrast, the brightness of the seam part is lower and the geometric features are not obvious, so mis-segmentation of the seam part is more likely to occur. Although we have thickened the line width of the seam portion of the label trained in the network, it is still impossible to completely avoid the mis-segmentation of the seam. To solve this problem, we adopt an image processing method based on the connected component analysis. Taking Figure 5 as an example, the upper edge of the groove portion is identified as the seam due to the similarity of the geometric features to the seam under low brightness. After analyzing the pixel distribution, it is found that the number of pixels in the mis-segmentation part is mostly discontinuous, and only the real seam part is more continuous. Based on this geometric feature, the connected domains in the image can be labeled, and the pixel score of each connected domain can be calculated. We select the connected domain with the highest score from all connected domains, mark it as the seam part, and set the pixel gray value for the rest of the connected domains to zero, as shown in Figure 6.

When calculating the offset, a point needs to be determined as the position of the welding torch. During the welding process, due to the deviation between the preset trajectory and the actual welding seam, the arc starting position will fall on the slope, and the arc shape at this time will be greatly deformed. If the geometric center of the arc part is simply selected as the position of the welding torch, this will cause certain errors. It can be found that the upper half of the arc part is relatively stable, and the error can be reduced by using the center position of the upper edge of the arc part as the position of the welding torch. For the seam part, the centerline of the segmented seam is extracted using the gray centroid method. Then, the least squares method is used to obtain the seam line equation. We bring the ordinate of the welding torch position into the equation to calculate the corresponding abscissa. By comparing it with the abscissa of the welding torch position, the final offset can be obtained, as shown in Figure 7.

4. Experiment and Analysis

In order to observe the effect of this method in welding seam tracking, we designed welding experiments for verification. The parameters of the welding experiments are shown in

Table 1. Experimental paraments.

Parament	Value
Welding speed	18 cm/min
Welding current	140 A
Wire feeding speed	1.4 m/min
Workpiece material	304 Stainless steel
Thickness of workpiece	5 mm
Shielding gas	Argon

.

In the welding experiments, the welding torch moves along the preset trajectory. V-groove splices with groove angles of 30 and 45 degrees are used, and the preset trajectory is set to be offset from the actual seam in the X direction. The specific teaching trajectory is divided into the following categories: no deviation from the actual seam, parallel to the actual seam and offset by 0.5 mm in the X direction, parallel to the actual seam and offset by 1 mm in the X direction, and offset 2 mm in the X direction. After arcing, the industrial camera will continuously image the arcing part, and the image information for different offset situations can be obtained.

During the welding process, the industrial camera is fixed on the welding torch, and the relative positions of the camera and the welding torch remain unchanged. Therefore, when calculating the offset between the welding seam and the welding torch, it is not necessary to convert the information from two-dimensional to three-dimensional, and it is only necessary to calculate the mapping relationship between the offset in the image and the actual offset. In the experiments, the field of view of the camera is small, and the calculation part is roughly in the center of the image, so the influence of the image distortion can be ignored. Due to the relative stillness of the camera and the welding torch, the pixel size can be measured as 0.06 mm/pixel by placing a ruler in the welding area.

In order to verify the superiority of the segmentation effect of BiseNetV2 with OHEM, this paper compares the training results of BiseNetV2 and ICNet with our method. The segmentation effect is shown in Figure 8, and the results of the intersection of union (IoU) are shown in

Table 2. Results of the intersection of union.

IoU	mIoU	IoU of arc	IoU of seam
BiseNetV2 with OHEM	75.3%	93.4%	57.2%
BiseNetV2	68.1%	95.0%	41.2%
ICNet	64.9%	95.2%	34.5%

.

It can be seen from the results in the table that the mIoU of BiseNetV2 with OHEM is higher than for BiseNetV2 and ICNet. Moreover, the mIoU of the seam part is much higher than for BiseNetV2 and ICNet. The reason is that the number of pixels occupied by the seam part in the collected images is much less than in the arc part, and it is difficult to extract the image features from the seam part. Therefore, when performing semantic segmentation, the difficult pixels are mainly concentrated in the seam part. For difficult pixels, the network parameters can be retrained using OHEM. In this way, the segmentation effect for the seam part can be improved. Using BiseNetV2 with OHEM can achieve about 57 FPS with pytorch1.7.0 and cuda110 in Nvidia RTX3090, which can achieve real-time segmentation.

According to the semantic segmentation results for BiseNetV2 with OHEM, we can calculate the offset between the welding seam and the welding torch according to the method in Section 3.3. The errors for the experimental results for each group are shown in

Table 3. Errors of prediction.

	Groove Angel 30		Groove Angel 45
offset	1 mm	2 mm	0 mm	0.5 mm
max_error	0.51 mm	0.47 mm	0.64 mm	0.39 mm
mean_error	0.09 mm	0.10 mm	0.07 mm	0.09 mm

. The specific offset prediction diagrams are shown in Figure 9.

Because the distance between the groove and the upper edge of the 45 degree groove splicing plate is small, if the preset offset is too large, the molten pool will not all fall on the groove. Therefore, the preset offset of the 45 degree splice plate is relatively small. It can be seen from the results in the table that the error tends to increase with the increase in the offset. The reason for this trend is that when the welding torch deviates from the seam, the brightness of the seam area will decrease, the feature extraction of the seam during the semantic segmentation will become more difficult, and the segmentation accuracy will decrease.

In order to calculate the offset of the welding seam and the welding torch more precisely, a filter method is added to predict the offset in this paper. After predicting the offset of the current frame, the mean and variance of the offset from the previous five frames are calculated. Here, we calculate the difference between the current frame offset and the mean, and compare this with the variance. If the difference between the two is large, the median filter is used. Otherwise, the mean filter is used.

The offset prediction results after filtering are shown in Figure 10, and the errors in each experiment are shown in

Table 4. Errors of prediction after filtering.

	Groove Angel 30		Groove Angel 45
offset	1 mm	2 mm	0 mm	0.5 mm
max_error	0.28 mm	0.35 mm	0.24 mm	0.30 mm
mean_error	0.06 mm	0.07 mm	0.04 mm	0.06 mm

.

It can be seen from Figure 10 and Table 4 that the average error of the prediction of the offset decreases by 0.03 mm after filtering. The maximum error is stable within 0.4 mm, which can meet the precision requirements for automatic welding.

5. Conclusions

In this paper, a passive vision welding seam tracking system based on the semantic segmentation method is proposed, and a series of experiments are designed to verify the effect of this method. The following conclusions can be drawn:

• In order to achieve high-efficiency feature extraction of the seam and the arc in the welding process, this paper proposes a feature extraction method based on deep learning. BiseNetV2 is used as the network architecture for semantic segmentation because of its accuracy and efficiency during segmentation. The network is trained with OHEM to improve the segmentation effect of the difficult-to-separate areas (mainly the seam part);
• It is verified that the segmentation effect of BiseNetV2 with OHEM is significantly better than that of BiseNetV2 and ICNet. In the designed feature extraction comparison experiment, the following results can be obtained using BiseNetV2 with OHEM. The IoU of the arc is 92.1%, the IoU of the seam is 55.1%, and the mIoU is 72.6%. Compared with ICNet and BiseNetV2, although the IoU of the arc part is slightly lower, the IoU of the seam part, which is more difficult to segment, is significantly higher and the mIoU is also higher;
• A filter method is used during offset prediction after the seam line fitting. The results of the welding experiments show that the average error of the offset predicted by the method in this paper is within 0.07 mm in the welding situation, and real-time segmentation of 57 FPS can be achieved, which can meet the requirements for high-precision real-time monitoring under actual welding conditions.

To sum up, the passive vision seam tracking system based on semantic segmentation proposed in this paper uses BiseNetV2 with OHEM to achieve end-to-end welding image segmentation rather than performing different methods on the seam and arc. Then, after removing the mis-segmented part, the offset is calculated. Finally, a filter method is proposed to obtain more precise tracking results, and the final prediction error is within 0.07 mm. With all the above methods, the welding seam tracking system can meet the precision and real-time requirements for automatic welding.