Object Segmentation by Spraying Robot Based on Multi-Layer Perceptron

Abstract

The vision system provides an important way for construction robots to obtain the type and spatial location information of the object. The characteristics of the construction environment, construction object, and robot structure are jointly examined in this paper to propose an approach of object segmentation by spraying the robot based on multi-layer perceptron. Firstly, the hand-eye system experimental platform is built through establishing the mathematical model of the system and calibrating the parameters of the model. Secondly, effort is made to carry out research on image preprocessing algorithms and related experiments, and compare the effects of different binocular stereo-matching algorithms in the actual engineering environment. Finally, research and an experiment are conducted to identify the applicability and effect of the depth image object segmentation algorithm based on multi-layer perceptron. The experimental results prove that the application of multi-layer perceptron to object segmentation by spraying robots can meet the requirement on solution accuracy and is suitable for the object segmentation of complex projects in real life. This approach not only overcomes the shortcomings of the existing recognition methods that are poor in accuracy and difficult to be used widely, but also provides basic data for the subsequent three-dimensional reconstruction, thus making a significant contribution to the research of image processing by spraying robots.

1. Introduction

The construction of the load-bearing structure of steel structure is one of the main schemes in the field of engineering construction either at present or for a long time in the future [1]. However, the materials of steel structures feature poor fire resistance. It is pressing to explore the use of robots to replace the original manual construction work in the fire protection spraying of steel structures, which has strict requirements on image processing by spraying robot and is of great significance to the automation, intelligence, less manned operation and even unmanned operation of the whole construction process.

To facilitate the effective image object segmentation by spraying robots, scholars have proposed a variety of models, such as grey models [2,3], deep learning neural networks [4,5], machine learning [6,7], grey time-varying parameters model [8], grey-neural network model [9,10], etc. Nevertheless, these models require time series monitoring data to be stable, face difficulties in capturing the relationship between nonlinear data, need a large-size sample of data, and have defects such as slow convergence speed or hardship to accurately determine the large number of parameters. In practice, the image data available to spraying robots is usually characterized by a small sample size and nonlinearity. It should be noted that depth images belong to gray images. In the case that the background image is complex, it is impossible to reliably finish the task of object segmentation through the traditional gray image segmentation method based on threshold or dynamic threshold. In recent years, intelligent optimization and machine learning algorithm, especially their combined models such as particle swarm optimization combined with support vector machine [11], integrated intelligent algorithm [12,13], genetic algorithm combined with support vector machine [14], neural network algorithm combined with machine learning model [15], have become the research focus of machine vision as a result of their excellent performance. Nonetheless, in the combined models above, intelligent optimization algorithm itself may be prone to issues such as low local optimal value and convergence accuracy. The combination of optimization algorithm and machine learning algorithm cannot reasonably determine the hyper-parameters of machine algorithm, thus leading to low prediction accuracy. Image object segmentation is now widely used in object detection [16]. Vision inspection by spraying robot is regarded as an image segmentation problem. For example, structural feature convolution neural network (SCNN) [17] uses an image segmentation model to segment lane lines, and adopts message passing and additional scene annotations to capture global context information to improve accuracy. It has stronger representation capability than traditional image processing methods, but intensive pixel-level communication requires a lot of computing resources, resulting in low processing efficiency of the algorithm. In Ref. [18], one pixel in each line of lane images is detected as lane line in the processing of these images. Compared with image segmentation algorithm, this method reduces the work of calculation and improves the reasoning speed, but it has low universality and cannot be used for lane line detection in multiple environments.

With the development of computer technology, computer hardware and software have been significantly improved in performance, creating an environment for the introduction and application of deep learning. As a method of machine learning, Artificial Neural Network (ANN) has been widely used in industrial fields such as intelligent buildings. Multi-Layer Perceptron (MLP) is a classical model of artificial neural network, which is developed from perceptron. Its main feature is that there are multiple neuron layers. The basic structure of MLP includes input layer, hidden layer and output layer. The number of hidden layers can be varied. The input layer to the hidden layer can be regarded as a fully connected layer, and the hidden layer to the output layer can be regarded as a classifier. Many recent studies on MLP [19] have shown that MLP can better extract global semantic information from images. Specifically, Cycle-MLP in Ref. [20] has achieved good results in the downstream tasks of computer vision such as image segmentation. In Ref. [21], the decoupling of training and reasoning is achieved through the technology of structural re-parameterization, where obvious improvement in accuracy is made without sacrificing the reasoning speed. For instance, the MLP model in Ref. [22] builds an internal group convolution layer while training to obtain local information, and combines with the technology of re-parameterization. This method has achieved better results in pattern recognition. MLP implementation form is to map multiple input data sets to a single output data set. The main advantage is that it solves the nonlinear problem that single-layer perceptron cannot solve on the basis of linear regression, which is favored by scholars in different fields. Zhang [23] used MLP model to predict the traffic flow of a certain road in Chongqing, and achieved good accuracy. Lyu et al. [24] focused on the application of multi-layer perceptron in the prediction of the growth of various microorganisms in aviation catering, and established a growth model for the microbial community. Ding Xuesong et al. [25] predicted protein denaturation temperature based on multi-layer perceptron model, which solved the shortcomings of traditional experimental methods. This data-driven method predicts by mining the implicit relationship between data and target state, which has low dependence on physical mechanism. It can directly use the original sampling data as input and extract important features for predicting response, which is more versatile.

Therefore, the region segmentation of the whole object is considered in this paper as the task of classifying each pixel point in the object. Multi-layer perceptron is used to classify each pixel point in the corrected, colored left-eye images, so as to complete the task of the object segmentation of steel structure columns. The segmentation results are employed to extract information from each depth image, and obtain the depth images containing only steel structure columns. This approach not only overcomes the shortcomings of the existing recognition methods that are poor in accuracy and difficult to be used widely, but also provides basic data for the subsequent three-dimensional reconstruction, thus making significant contribution to the research of image processing by spraying robot.

2. Structure and Modeling of the System

2.1. Construction of the System Experimental Platform

The main hardware of the system experimental platform is shown in Figure 1. In order to reduce the movement of the robot body in the spraying process, a six-axis mechanical arm is used to meet the flexibility requirement for the planning of spraying trajectory. The maximum arm span of the mechanical arm is 1.710 m, and the end pose accessibility is good within 1.5 m; the rated load of the wrist is 20 kg, and the repeated positioning accuracy at the end space is ±0.1 mm, meeting the load requirements of end devices such as the spray gun and the binocular camera. A monocular 1080P high-resolution CMOS sensor is used as the binocular camera. The lens field angle and the focal length are estimated according to the end reach range of the mechanical arm in the horizontal and vertical directions. The standard for fine adjustment of the binocular focal length is determined to be the clear imaging of the object at 1.4 m away from the camera.

2.2. Mathematical Model of the Robot’s Hand-Eye System

2.2.1. Theory of Coordinate System Transformation

Without losing generality, $O_{\mathrm{A}}-X_{\mathrm{A}}{Y}_{\mathrm{A}}{Z}_{\mathrm{A}}$ and $O_{\mathrm{B}}-X_{\mathrm{B}}{Y}_{\mathrm{B}}{Z}_{\mathrm{B}}$ are respectively set to be two Cartesian space rectangular coordinate systems. The coordinates of the point P in space in the coordinate system and {B} are $P_{\mathrm{A}}\left(x_{\mathrm{A}},y_{\mathrm{A}},z_{\mathrm{A}}\right)$ and $P_{\mathrm{B}}\left(x_{\mathrm{B}},y_{\mathrm{B}},z_{\mathrm{B}}\right)$ respectively. The coordinate system {B} can be obtained from the coordinate system {A} through the following transformation process: rotating in the positive direction around the axis $Z_{\mathrm{A}}$ by the angle $\gamma$, then rotating in the positive direction around the axis $Y_{\mathrm{A}}$ by the angle $\beta$ and then rotating in the positive direction around the axis $X_{\mathrm{A}}$ by the angle $\alpha$, finally with the origin $O_{\mathrm{B}}$ translating upwards in the positive direction at$X_{\mathrm{A}}$, $Y_{\mathrm{A}}$ and $Z_{\mathrm{A}}$ respectively by $t_X$, $t_Y$ and $t_Z$. According to the theory of spatial coordinate system transformation [25], it can be obtained that:

$$\left[\begin{array}{c}{x}_{\mathrm{A}}\\ y_{\mathrm{A}}\\ z_{\mathrm{A}}\end{array}\right]=R_X\left(\alpha \right)R_Y\left(\beta \right)R_Z\left(\gamma \right)\left[\begin{array}{c}{x}_{\mathrm{B}}\\ y_{\mathrm{B}}\\ z_{\mathrm{B}}\end{array}\right]+\left[\begin{array}{c}{t}_X\\ t_Y\\ t_Z\end{array}\right]\triangleq {}_{\mathrm{B}}{}^{\mathrm{A}}R\left[\begin{array}{c}{x}_{\mathrm{B}}\\ y_{\mathrm{B}}\\ z_{\mathrm{B}}\end{array}\right]+{}_{\mathrm{B}}{}^{\mathrm{A}}T$$

(1)

where, the matrix ${}_{\mathrm{B}}{}^{\mathrm{A}}R$ is the rotation matrix from the coordinate system {B} to the coordinate system {A}, and the matrix ${}_{\mathrm{B}}{}^{\mathrm{A}}T$ is the translation matrix from the coordinate system {B} to the coordinate system {A}.$R_X\left(\alpha \right)$, $R_Y\left(\beta \right)$, $R_Z\left(\gamma \right)$ and ${}_{\mathrm{B}}{}^{\mathrm{A}}R$ are all invertible matrices and are respectively expressed as:

$$R_X\left(\alpha \right)=\left[\begin{array}{ccc}1& 0& 0\\ 0& \cos \alpha & -\sin \alpha \\ 0& \sin \alpha & \cos \alpha \end{array}\right]$$

(2)

$$R_Y\left(\beta \right)=\left[\begin{array}{ccc}\cos \beta & 0& \sin \beta \\ 0& 1& 0\\ -\sin \beta & 0& \cos \beta \end{array}\right]$$

(3)

$$R_Z\left(\gamma \right)=\left[\begin{array}{ccc}\cos \gamma & -\sin \gamma & 0\\ \sin \gamma & \cos \gamma & 0\\ 0& 0& 1\end{array}\right]$$

(4)

$${}_{\mathrm{A}}{}^{\mathrm{B}}R=R_X\left(\alpha \right)R_Y\left(\beta \right)R_Z\left(\gamma \right)\triangleq \left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\\ r_{31}& r_{32}& r_{33}\end{array}\right]$$

(5)

The rotation angle $\alpha$, $\beta$ and $\gamma$ can be obtained by the element $r_{ij}$ in the matrix ${}_{\mathrm{A}}{}^{\mathrm{B}}R$ through calculation according to Equation (6):

$$\{\begin{array}{c}\alpha =\mathrm{atan}2\left(r_{32},r_{33}\right)\\ \beta =\mathrm{atan}2\left(-r_{31},\sqrt{r_{31}{}^2+r_{33}{}^2}\right)\\ \gamma =\mathrm{atan}2\left(r_{21},r_{11}\right)\end{array}$$

(6)

To facilitate continuous coordinate transformation, Equation (1) is extended to homogeneous forms, including:

$$\left[\begin{array}{c}{x}_{\mathrm{A}}\\ y_{\mathrm{A}}\\ z_{\mathrm{A}}\\ 1\end{array}\right]=\left[\begin{array}{cc}{}_{\mathrm{B}}{}^{\mathrm{A}}R& {}_{\mathrm{B}}{}^{\mathrm{A}}T\\ 0& 1\end{array}\right]\left[\begin{array}{c}{x}_{\mathrm{B}}\\ y_{\mathrm{B}}\\ z_{\mathrm{B}}\\ 1\end{array}\right]\triangleq {}_{\mathrm{B}}{}^{\mathrm{A}}M\left[\begin{array}{c}{x}_{\mathrm{B}}\\ y_{\mathrm{B}}\\ z_{\mathrm{B}}\\ 1\end{array}\right]$$

(7)

where, the matrix ${}_{ \mathrm{B}}{}^{\mathrm{A}}M$ is the transformation matrix from the coordinate system {B} to coordinate system {A}, and there is:

$${\left({}_{\mathrm{B}}{}^{\mathrm{A}}M\right)}^{-1}={\left[\begin{array}{cc}{}_{\mathrm{B}}{}^{\mathrm{A}}R& {}_{\mathrm{B}}{}^{\mathrm{A}}T\\ 0& 1\end{array}\right]}^{-1}=\left[\begin{array}{cc}{}_{\mathrm{B}}{}^{\mathrm{A}}R{}^{-1}& -{}_{\mathrm{B}}{}^{\mathrm{A}}T\\ 0& 1\end{array}\right]=\left[\begin{array}{cc}{}_{\mathrm{A}}{}^{\mathrm{B}}R& {}_{\mathrm{A}}{}^{\mathrm{B}}T\\ 0& 1\end{array}\right]={}_{\mathrm{A}}{}^{\mathrm{B}}M$$

(8)

2.2.2. Eye-in-Hand Model

In the hand-eye system, the binocular camera is installed at the end of the sixth axis of the mechanical arm through a fixing bracket. The Cartesian space rectangular coordinate system of the system is established in accordance with the way shown in Figure 2. Specifically, $O_{\mathrm{base}}-X_{\mathrm{base}}{Y}_{\mathrm{base}}{Z}_{\mathrm{base}}$is the coordinate system of the base of the mechanical arm and also the world coordinate system, of which the origin is located in the center of the base of the mechanical arm; $O_{\mathrm{tool}}-X_{\mathrm{tool}}{Y}_{\mathrm{tool}}{Z}_{\mathrm{tool}}$ is the tool coordinate system, of which the origin is located in the center of the flange plate of the sixth axis of the mechanical arm; $O_{\mathrm{cam}}-X_{\mathrm{cam}}{Y}_{\mathrm{cam}}{Z}_{\mathrm{cam}}$ is the ideal binocular camera coordinate system, of which the origin is located at the optical center of the left-eye camera; $O_{\mathrm{cal}}-X_{\mathrm{cal}}{Y}_{\mathrm{cal}}{Z}_{\mathrm{cal}}$ is the calibration plate coordinate system, of which the origin is located in the center of the calibration plate.

2.3. Mathematical Model of the Binocular Camera

The coordinate system is constructed according to the way shown in Figure 3, in order to analyze and establish the imaging mathematical model of the binocular camera used in this study.

Where, $O_{\mathrm{Lc}}-X_{\mathrm{Lc}}{Y}_{\mathrm{Lc}}{Z}_{\mathrm{Lc}}$ is the left-eye camera coordinate system, with $O_{\mathrm{Lc}}$ being located at the optical center of the left-eye camera and $Z_{\mathrm{Lc}}$ being the optical axis of the left-eye camera; $O_{\mathrm{Rc}}-X_{\mathrm{Rc}}{Y}_{\mathrm{Rc}}{Z}_{\mathrm{Rc}}$ is the right-eye camera coordinate system, with$O_{\mathrm{Rc}}$ being located at the optical center of the right-eye camera and $Z_{\mathrm{Rc}}$ being the optical axis of the right-eye camera; the distance between $O_{\mathrm{Rc}}$ and $O_{\mathrm{Lc}}$ is b, that is, the baseline length; the matrix ${}_{\mathrm{Rc}}{}^{\mathrm{Lc}}M$ is the transformation matrix from the coordinate system to the coordinate system {Lc}, and the matrix ${}_{\mathrm{Lc}}{}^{\mathrm{base}}M$ is the transformation matrix from the coordinate system {Lc} to the coordinate system {base}; $O_{\mathrm{Lp}}-U_{\mathrm{Lp}}{V}_{\mathrm{Lp}}$ and $O_{\mathrm{Rp}}-U_{\mathrm{Rp}}{V}_{\mathrm{Rp}}$ are the left- and right-eye pixel coordinate system respectively; $O_{\mathrm{Li}}-X_{\mathrm{Li}}{Y}_{\mathrm{Li}}$ and $O_{\mathrm{Ri}}-X_{\mathrm{Ri}}{Y}_{\mathrm{Ri}}$ are the left- and right-eye image coordinate system respectively; The coordinate of $O_{\mathrm{Li}}$in the coordinate system {Lp} is $\left(u_{\mathrm{L}0},v_{\mathrm{L}0}\right)$ and its distance from $O_{\mathrm{Lc}}$ is the left-eye lens focal length $f_{\mathrm{Lc}}$; The coordinate of$O_{\mathrm{Ri}}$ in the coordinate system is$\left(u_{\mathrm{R}0},v_{\mathrm{R}0}\right)$ and its distance from $O_{\mathrm{Rc}}$ is the right-eye lens focal length $f_{\mathrm{Rc}}$; the images of the Point $P_{\mathrm{base}}$ in space in the left and right imaging planes are respectively $P_{\mathrm{Li}}$ and $P_{\mathrm{Ri}}$.

3. Calibration of System Parameters

3.1. Calibration of Binocular Camera Parameters

The binocular calibration task is to determine the accurate values of relevant parameters in the camera model, covering the camera focal length$f_{\mathrm{c}}$, the pixel sizes including $S_U$ and$S_V$, the pixel coordinate of the image center point $\left(u_0,v_0\right)$, the imaging distortion model parameters including $k_1$, $k_2$, $k_3$, $p_1$ and $p_2$, and the transformation matrix ${}_{ \mathrm{Rc}}{}^{\mathrm{Lc}}M$ from the coordinate system to the coordinate system {Lc} including the rotation matrix ${}_{\mathrm{Rc}}{}^{\mathrm{Lc}}R$ and the translation matrix ${}_{\mathrm{Rc}}{}^{\mathrm{Lc}}T$.

The binocular calibration experiment in this paper adopts high-precision 7 × 7 center displaying calibration plate. The calibration pattern region is a square with a side length of 400 mm, in which the circular pattern has a diameter of 25 mm and a spacing of 50 mm. There is a pose identification mark of the calibration plate coordinate system {cal} in the upper left corner. The binocular camera calibration experimental equipment and layout are shown in Figure 4.

The binocular camera calibration results obtained by the calibration plate through image acquisition and landmark extraction are shown in

Table 1. Calibration results of internal parameters of binocular camera.

Parameter	${\mathit{f}}_{\mathit{c}}$/mm	${\mathit{S}}_{\mathit{U}}$/μm	${\mathit{S}}_{\mathit{V}}$/μm	${\mathit{u}}_0$	${\mathit{v}}_0$	Image Width	Image High
Left eye	4.767	3.394	3.380	967.551	529.788	1920	1080
Right eye	4.776	3.394	3.380	951.657	540.111	1920	1080

,

Table 2. Distortion parameter calibration results.

Parameter	${\mathit{k}}_1$/(m2)−1	${\mathit{k}}_2$/(m4)−1	${\mathit{k}}_3$/(m6)−1	${\mathit{p}}_1$/(m2)−1	${\mathit{p}}_2$/(m2)−1
Left eye	−519.880	−1.594 ×107	4.256 × 1012	0.124104	−0.236724
Right eye	−482.356	−3.230 ×107	5.406 × 1012	0.180746	−0.204752

and

Table 3. Calibration results of binocular position relationship parameters.

Parameter	$\mathbf{Rotation} \mathbf{Matrix} {}_{\mathbf{R}\mathbf{c}}{}^{\mathbf{L}\mathbf{c}}\mathit{R}$			$\mathbf{Translation} \mathbf{Vector} {}_{\mathbf{R}\mathbf{c}}{}^{\mathbf{L}\mathbf{c}}\mathit{T}$
	${\mathit{\alpha }}_{\mathbf{L}\mathbf{c}}$/deg	${\mathit{\beta }}_{\mathbf{L}\mathbf{c}}$/deg	${\mathit{\gamma }}_{\mathbf{L}\mathbf{c}}$/deg	${\mathit{X}}_{\mathbf{L}\mathbf{c}}$/mm	${\mathit{Y}}_{\mathbf{L}\mathbf{c}}$/mm	${\mathit{Z}}_{\mathbf{L}\mathbf{c}}$/mm
parameter value	0.0238	0.2601	359.9840	60.1305	0.0956	1.1106

.

The average double projection error of binocular calibration is 0.1104 pixels, and the overall calibration error is small. The values of $k_1$, $k_2$ and $k_3$ are relatively large, because $r_{\mathrm{Li}}$ and $r_{\mathrm{Ri}}$ in the distortion model are taken in meters, while the absolute values of their actual physical quantities are quite small. At the same time, the values of $p_1$ and $p_2$ are smaller, indicating that there is less tangential distortion in the imaging process and that the imaging distortion is dominated by radial distortion.

3.2. Calibration of Hand-Eye Positional Relationship Parameters

The goal of hand-eye calibration is to obtain the positional relationship between the ideal binocular camera coordinate system {cam} and the tool coordinate system {tool} (or the base coordinate system {base} of the mechanical arm). There is no essential difference between the two hand-eye system calibration methods. With the assistance from HALCON algorithm library, the basic process of hand-eye calibration in this study is:

(1)

Adjusting the end pose of the mechanical arm using a teach pendant or remote-control mode, and reading the pose of the end tool coordinate system {tool} under the coordinate system {base} from the information system of the mechanical arm;

(2)

Using the left-eye camera to capture the images containing the complete calibration plate and adopting the operator map_image() to correct the correction mapping images (file of maps) obtained from calibration by the binocular camera;

(3)

Using the operator find_calib_object() to find the object in the corrected calibration plate images and adopting the operator get_calib_data_observ_pose() to extract the pose of the calibration plate coordinate system {cal} under the camera coordinate system {cam};

(4)

Repeating the process above to obtain 15 sets of pose relationships, using calibrate_hand_eye() to perform calibration calculation according to the hand-eye system model and using get_calib_data() to read the results.

4. Binocular Stereo Matching

4.1. Image Preprocessing

Because the actual binocular camera objectively has some problems such as imaging distortion, sensor noise and non-ideal parameters of the binocular model, it is necessary to correct, filter and gray the original-colored binocular images in preprocessing before the stereo matching of binocular images, so that they can meet the basic requirement for binocular stereo matching. In addition to some binocular images collected in the laboratory environment for verifying the algorithm’s accuracy, the binocular images used in the actual effect verification of the algorithm are taken from the construction site. The measured object is large-scale square steel structure columns. During the acquisition of images, the binocular camera keeps the optical axis horizontal and is fixed at the position 1.565 m in front of the rectangular edge of steel structure columns (the horizontal distance between the optical center of the left-eye camera and the rectangular edge is measured by a laser rangefinder). The acquired binocular images are shown in Figure 5.

4.1.1. Smoothing Filtering

In the process of image acquisition and signal transmission by the camera, its electronic components would inevitably be subject to external electromagnetic interference, so there is often noise in the image obtained. In order to reduce the effect of imaging noise on stereo matching, it is necessary to eliminate noise through image smoothing filtering. In this study, Gaussian filtering is used as the image smoothing filtering method. For the Gaussian template $H$ with a side length of 2k + 1, if the center point coordinate of the template is set to (0,0), the formula for calculating each element in the template is:

$$H_{\left(i,j\right)}=\frac{1}{2\pi {\sigma }^2}\exp \left[-\frac{{\left(i-k-1\right)}^2+{\left(j-k-1\right)}^2}{2{\sigma }^2}\right], i,j\in \left[-k,k\right]$$

(9)

where, $k$ and $\sigma$ are the parameters to be designed in the template. In practical application, the template size $k$ and the parameters $\sigma$ need to be selected and adjusted according to the actual image filtering effect.

4.1.2. Image Graying

Image graying is the process of transforming RGB three-channel (or multi-channel) images into single-channel images, to realize the compression of image information and increase the contrast ratio.

The weighted average method is a commonly used approach of graying. This method assigns different weights to different channels according to the different sensitivity of human eyes to different colors, so that the gray images obtained are closer to the subjective feelings of human beings. In this study, the weighted average method is used as the approach of image graying. The formula for calculating each pixel point in the gray images is as follows:

$$I\left(u,v\right)=0.299\ast R\left(u,v\right)+0.587\ast G\left(u,v\right)+0.114\ast B\left(u,v\right)$$

(10)

where, $R\left(u,v\right)$, $G\left(u,v\right)$ and $B\left(u,v\right)$are respectively the pixel values of the image Red channel, Green channel and Blue channel at the pixel point (u, v).

4.1.3. Histogram Equalization

Due to the complex lighting conditions in the construction environment, especially the uncontrollable and uneven illumination of natural light on the object, the difference in the angle of view of the binocular camera in this environment would cause a certain brightness difference in the collected binocular images, which destroys the basic assumption that stereo matching algorithm depends on. In order to reduce the impact of the brightness difference in binocular images on subsequent stereo matching, this paper uses the method of histogram equalization to equalize the histograms of binocular gray images and thus to balance the dynamic brightness range of the binocular images.

The basic idea of histogram equalization is to transform the histogram of the original gray image into a form of uniform distribution in the whole gray range. If the total number of pixel points in the image is n and the gray value of pixel points is $k\in \left[0, 255\right]$, the corresponding histogram equalization process is:

(1)

Traversing the whole image to count the number $n_k$of pixel points with the gray value $I\left(u,v\right)=k$;

(2)

Calculating the probability of pixels with each gray value in the image:

$$P\left(I_k\right)=\frac{n_k}{n}$$

(11)

(3)

For the pixel points with $I\left(u,v\right)=k$ in the original gray image, calculating the gray value of each pixel point after equalization:

$$I^{\prime }\left(i,j\right)=\mathrm{int}\left(255\ast {\displaystyle \sum }_{t=0}^{k}P\left(I_t\right)+0.5\right)=\mathrm{int}\left(255\ast {\displaystyle \sum }_{t=0}^k\frac{n_t}{n}+0.5\right)$$

(12)

where,$\mathrm{int}()$ is takes the lower integer ceiling function. After the steps above are finished, the dynamic brightness range of the left and right images is balanced to the range of [0, 255]. In this way, the overall brightness difference in gray images is reduced to a certain extent.

4.1.4. Image Preprocessing Experiment

The image preprocessing experiment is completed under the HALCON development environment. For the binocular images obtained (Figure 5), gauss_image() is first used to smooth the binocular images. The side length of the Gaussian template is 3, and σ is set to 0.6. The filtered binocular images are shown in Figure 6.

Afterwards, the operator rbg1_to_gray() is used to convert the filtered colored binocular images into the corresponding gray images, as shown in Figure 7a,b.

Finally, the operator equ_histo_image() is used to perform histogram equalization. The gray level histograms of the left-eye and right-eye images before equalization are shown in Figure 8a,b, with the abscissa being $k$ and the ordinate being$n_k$; the equalized gray histograms are shown in Figure 8c,d, and the equalized gray images are shown in Figure 8e,f.

It can be seen from the histogram equalization results that compared with the gray images before equalization, the gray images after equalization, on the one hand, have a wider dynamic range, more even distribution of pixel gray values and a larger gray gap between pixel points with similar gray values in the original images, which is conducive to subsequent binocular stereo matching. On the other hand, histogram equalization also causes discontinuous changes in the gray values of the images and reduction of gray levels. The number of gray levels in Figure 8e is 28.17% less than that in Figure 7a, and the number of gray levels in Figure 8f is 26.98% less than that in Figure 7b, which is unfavorable to binocular stereo matching to some extent. Nevertheless, for gray images with unbalanced distribution of the brightness dynamic range and binocular brightness difference, histogram equalization has a positive effect on subsequent binocular stereo matching.

4.2. Binocular Stereo Matching

4.2.1. Parallax and Depth

According to the equivalent ideal binocular camera parameters of the binocular imaging model, the point $P_{\mathrm{Lc}}\left(x_{\mathrm{Lc}},y_{\mathrm{Lc}},z_{\mathrm{Lc}}\right)$ under the left-eye camera coordinate system forms the projection points $P_{\mathrm{Lp}}\left(u_{\mathrm{Lp}},v_{\mathrm{Lp}}\right)$ and $P_{\mathrm{Rp}}\left(u_{\mathrm{Rp}},v_{\mathrm{Rp}}\right)$ respectively on the left and right imaging planes. The simplified model is shown in Figure 9.

The corresponding coordinate transformation relationship in the imaging process is:

$$z_{\mathrm{Lc}}\left[\begin{array}{c}{u}_{\mathrm{Lp}}\\ v_{\mathrm{Lp}}\\ 1\end{array}\right]=\left[\begin{array}{cccc}\frac{f_{\mathrm{c}}}{S_U}& 0& u_{\mathrm{L}0}& 0\\ 0& \frac{f_{\mathrm{c}}}{S_V}& v_{\mathrm{L}0}& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}{x}_{\mathrm{Lc}}\\ y_{\mathrm{Lc}}\\ z_{\mathrm{Lc}}\\ 1\end{array}\right]$$

(13)

$$z_{\mathrm{Lc}}\left[\begin{array}{c}{u}_{\mathrm{Rp}}\\ v_{\mathrm{Rp}}\\ 1\end{array}\right]=\left[\begin{array}{cccc}\frac{f_{\mathrm{c}}}{S_U}& 0& u_{\mathrm{R}0}& 0\\ 0& \frac{f_{\mathrm{c}}}{S_V}& v_{\mathrm{R}0}& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}{x}_{\mathrm{Lc}}\\ y_{\mathrm{Lc}}\\ z_{\mathrm{Lc}}\\ 1\end{array}\right]$$

(14)

As a result of binocular correction,$\mathsf{\Delta }{P}_{\mathrm{Lc}}{P}_{\mathrm{Li}}{P}_{\mathrm{Ri}}~\mathsf{\Delta }{P}_{\mathrm{Lc}}{O}_{\mathrm{Lc}}{O}_{\mathrm{Rc}}$ is obtained. In combination with Equations (4) and (5) and Equations (4)–(6) and based on the coordinate $\left(u_{\mathrm{Lp}},v_{\mathrm{Lp}}\right)$of $P_{\mathrm{Lp}}$ in the left-eye images, it can be obtained that:

$$z_{\mathrm{Lc}}=\frac{b{f}_c}{S_U\left(u_{\mathrm{Lp}}-u_{\mathrm{Rp}}-u_{\mathrm{L}0}+u_{\mathrm{R}0}\right)}\triangleq \frac{b{f}_c}{S_U\left(d-d^{\prime }\right)}$$

(15)

$$x_{\mathrm{Lc}}=\frac{S_U\left(u_{\mathrm{Lp}}-u_{\mathrm{L}0}\right)}{f_{\mathrm{c}}}\left(z_{\mathrm{Lc}}-f_{\mathrm{c}}\right)$$

(16)

$$y_{\mathrm{Lc}}=\frac{S_V\left(v_{\mathrm{Lp}}-v_{\mathrm{L}0}\right)}{f_{\mathrm{c}}}\left(z_{\mathrm{Lc}}-f_{\mathrm{c}}\right)$$

(17)

where, $z_{\mathrm{Lc}}$ is also called the depth value ($Depth$) of the space point $P$ under the camera coordinate system {Lc}; $b$ is the baseline distance of the binocular camera after binocular correction; $d=u_{\mathrm{Lp}}-u_{\mathrm{Rp}}$ is the parallax of the binocular images; $d^{\prime }=u_{\mathrm{L}0}-u_{\mathrm{R}0}$ is the compensation item of parallax, which is a constant related to the principal point coordinate of the corrected binocular camera.

4.2.2. Binocular Stereo Matching Experiment

Binocular stereo matching is the antecedent link of restoring the depth information of space points. Stereo matching algorithm is used to find the matching relationship between the corresponding projection points of space points on the left and right imaging planes of the binocular camera, and the parallax of the corresponding points is calculated based on the left-eye image coordinate system. Then, the corresponding depth images are calculated from the parallax images.

Normalized Cross Correlation (NCC) algorithm uses the average gray level of pixels in the matching window to normalize the gray value of each point in the matching window, thus realizing the compensation for the brightness difference in the matching window. Multi-grid method takes the global energy function as the core, and the components of the global energy function take into account the gray level, the gray gradient and the impact of too large parallax changes. The information of images is used more fully, and the parallax images obtained are more continuous and smoother. This paper compares the local stereo matching algorithm based on NCC with the global stereo matching algorithm based on multi-grid method.

In the construction site environment, steel structure columns are taken as the foreground object, and the camera is facing the rectangular edge of steel structure columns. The horizontal distance between the optical center of the left-eye camera and the rectangular edge of steel structure columns measured by the laser rangefinder is 1.565 m. The algorithm parameters set in this experiment are consistent with those set in the in-laboratory book ranging experiment. The original images and the parallax images obtained through calculation are shown in Figure 10.

In regions of steel structure columns shown in Figure 10, because of mismatches by NCC algorithm in the repeated texture regions, there are many parallax holes in the parallax images after the elimination of mismatches. In contrast, the parallax obtained by multi-grid method is uniform and continuous, which is consistent with the actual change in the depth of steel structure columns. The middle of the rectangular edge of steel structure columns is selected as the point for depth measurement. The depth obtained by multi-grid method through visual measurement is 1.548 m, compared with the actual depth of 1.565 m, with the relative error being −1.09% and the absolute error being 0.017 m, meeting the requirement of spraying operation on visual measurement accuracy.

5. Depth Image Object Segmentation

5.1. Workflow and Structure of Multi-Layer Perceptron

Multi-layer perceptron (MLP) is a kind of artificial neural network that has several advantages including a simple structure, intuitive model parameters, easy adjustment and fast model training speed. MLP can achieve better results for the task of pixel classification simple and fixed scenes. However, due to the complexity of the actual construction environment, there may be great changes in the colored images taken at different time periods and different operation positions. It is difficult for MLP to collect training data covering all working conditions. Besides, the pixel classification ability of a single MLP cannot cover all possible construction conditions. To solve the problems above, the method adopted in this paper is to use the color difference between the foreground object and the background image. According to the real-time object image, the assistant operator of the robot selects the local regions of the object in the box on the operation screen as the training set, followed by model training and object segmentation under this data set. The workflow is shown in Figure 11.

The MLP network is trained using BP algorithm. Its network structure and the rationale of object segmentation are shown in Figure 12: First, the left-eye colored images are spilt into image channels to obtain R, G and B channel images respectively; Then, for a pixel point in an colored image to be classified, the gray value of the pixel point at the corresponding position in R, G and B channels is taken as the input of MLP, and the network output after classification judgment and mapping is the category the pixel belongs to; the steps above are repeated until the classification of all pixel points in the colored images is completed.

The neuron node model is shown in Figure 13:

The input-output relationship of the$k$-th neuron node is:

$$ne{t}_k={\displaystyle \sum }_{i=1}^n{w}_{ki}{x}_{ki}-{\theta }_k$$

(18)

$$y_k=f_k\left(ne{t}_k\right)$$

(19)

where $n$ is the number of the input items of the neuron node; $x_{ki}$ is each input item; $w_{ki}$ is the weight value corresponding to the input item; ${\theta }_k$ is the activation threshold of the neuron node; $f_k$is the activation function; $y_k$ is the output value of the neuron node.

For neuron nodes in the hidden layer, $f_k$ takes the tanh function as the activation function:

$$f_k=\tanh \left(ne{t}_k\right)=\frac{\exp \left(ne{t}_k\right)-\exp \left(-ne{t}_k\right)}{\exp \left(ne{t}_k\right)+\exp \left(-ne{t}_k\right)}$$

(20)

For neural nodes in the output layer, in order to carry out the normalization operation to obtain classification results in the form of probability, $f_k$ takes the softmax function as the activation function:

$$f_k=\mathrm{softmax}\left[\exp \left(ne{t}_k\right)\right]=\frac{\exp \left(ne{t}_k\right)}{{{\displaystyle \sum }}_{i=1}^j\exp \left(ne{t}_i\right)}$$

(21)

where $j$ is the number of neuron nodes in the output layer, that is, the number of categories. The output value of nodes in the form of probability contributes to more intuitive classification judgment, that is, the classification corresponding to the node with the highest probability value is taken as the category of the pixel.

In this paper, the cross entropy loss function is used as the loss function of the neural network. Compared with the classification error rate loss function and the mean square error loss function, the network error value given by the cross entropy loss function is more stable. The cross entropy loss function is defined as follows:

$$Loss=-\frac{1}{N}{\displaystyle \sum }_i{\displaystyle \sum }_{c=1}^j{\widehat{p}}_{ic}\log \left(p_{ic}\right)$$

(22)

where ${\widehat{p}}_{ic}$ is the sign function, which takes 1 when the real category of the sample i is consistent with the given category c and takes 0 otherwise; $p_{ic}$ is the predicted probability that the observation sample i belongs to the category c.

In the MLP network structure above, the number of nodes in the input layer is fixed to 3, and the number of nodes in the output layer is set according to the type of the object to be classified. The number of nodes in the hidden layer has great impact on the accuracy of pixel classification, the generalization performance of the neural network, and the computational resources consumed by neural network training. This parameter needs to be adjusted according to the actual pixel classification effect.

5.2. Object Segmentation Experiment

(1)

Creation of the Training Set and Neural Network

The training set is generated by manually aided frame selection and label setting, as shown in Figure 14.

During the creation of the network training set, the regions shown in the label of “Steel Structure Columns” label are assigned a separate category, and the regions shown in the labels “Ground” and “Background” are both classified as “Other”. After setting the regions and labels, the operator add_samples_image_class_mlp() is used to generate the training set of the MLP network.

The MLP network is created through the operator create_class_mlp(), of which the main parameters are:

(1)

The number of nodes in the input layer is 3; the number of nodes in the hidden layer is 6; the number of nodes in the output layer is 2;

(2)

The tanh function is the activation function of the hidden layer;

(3)

The softmax function is the activation function of the output layer.

The MLP network is trained through the operator train_class_mlp(). The main parameters of the operator are:

(1)

The number of training iterations is 400;

(2)

The threshold of weight change is 1;

(3)

The threshold of iteration error is 0.1.

The change in the number of misclassified points during training is shown in Figure 15.

In Figure 15, the training sample includes a total number of 122,791 pixel points. The number of misclassified points in the sample is reduced to 6176 after the 200th iteration, and is 5564 after the 400th iteration. In the first 25 iterations, the number of classified pixel points decreases rapidly and converges. When the iteration reaches about 200 times, the changing trend of the number of misclassified points is basically stable. The number of misclassified points after 400 iterations accounts for only 4.53% of the training sample.

(2)

Object Segmentation

After the MLP network training is completed, the operator classify_image_class_mlp() is used to perform object segmentation on the original images. The segmentation results of the original object are shown in the yellow region in Figure 16.

It can be seen from Figure 16 that the MLP network has a good segmentation effect on the object overall. The main region of steel structure columns has clear edges and accurate segmentation, and there are only sporadic small holes in the region; There are some misclassified pixel points outside the main region, such as in the upper left corner region and the right-side region of the images. Because of the small number of mismatched points, it is possible to process the mismatched points according to regional morphology.

(3)

Regional Morphologic Processing

The steps of regional morphologic processing adopted in this paper are as follows:

(1)

the operator fill_up() is used to fill the holes in the regions;

(2)

the operator opening_rectangle1() is used with a rectangular template to perform an opening operation on the regions and eliminate isolated points, burrs on regional edges and narrow bridges connecting the regions;

(3)

the operator select_shape() is used to select regions with the pixel area of the region as the filtering element and retain the regions with a large pixel area;

(4)

the operator closing_rectangle1() is used with a rectangular template to perform a closed operation on the regions, fill in notches on region edges, and make region edges flat.

The regions in Figure 16 are processed according to regional morphology, and the results are shown in Figure 17.

It can be seen from Figure 17 that the method of regional morphologic processing adopted in this paper eliminates a small number of original mismatched points, and fill the internal holes in the main region of steel structure columns. The amount of data is greatly reduced while the necessary information for the 3D reconstruction of the object is retained, providing basic data for subsequent surface 3D reconstruction of steel structure columns.

6. Conclusions

This paper examines the method of object segmentation based on MLP, and gives the specific workflow that can realize object segmentation in practical projects. This approach not only overcomes the shortcomings of the existing recognition methods that are poor in accuracy and difficult to be used widely, but also provides basic data for the subsequent three-dimensional reconstruction, thus making a significant contribution to the research of image processing by spraying robots. The experimental results show that the MLP network, in combination with simple manual assistance to delimit the training sample and necessary regional morphological processing, has a better object segmentation effect in local fixed scenes. Meanwhile, MLP is used to segment the position of the target region in the left-eye-colored images, realizing the task of object segmentation of the depth images.

However, in the actual recognition and characterization process, it is necessary to select the artificial intelligence algorithm which is compatible with the data distribution characteristics in order to achieve the best recognition effect. Subsequent work can be considered to further improve the algorithm so that it can be applied to steel structures, and can shorten the training time.