Panoramic UAV Image Mosaic Method and Its Application in Pavement Paving Temperature Monitoring

Abstract

The low-altitude technology of unmanned airborne infrared detection system is used to effectively monitor the temperature segregation in the paving stage and realize the temperature uniformity control of asphalt pavement construction. The image mosaic method can splice two images with overlapping areas together to form a panoramic image. In order to solve the problems of long time-consuming and low accuracy of aerial image mosaic algorithm, the low-temperature area of the whole pavement can be obtained quickly and accurately. In this paper, threshold segmentation technology is introduced to convert the image captured by the unmanned aerial vehicle (UAV) into a binary greyscale image so as to compensate for the mosaic error caused by temperature difference. In order to improve the efficiency and accuracy of splicing, a reference plate is used, which can provide enough feature points for splicing. Finally, the image mosaic method proposed in this paper can quickly obtain the image of the whole low-temperature area of the newly paved asphalt pavement, which has practical value and positive significance for the quality control of asphalt pavement.

1. Introduction

The paving construction of hot mix asphalt pavement requires a very high temperature. From the mixing, transportation, paving to rolling of asphalt mixture, temperature control directly affects the quality and service life of pavement entities. During asphalt pavement paving, temperature segregation will seriously affect the pavement quality. It is of great significance to quickly and accurately obtain the low-temperature area of the entire pavement width for the evaluation of asphalt pavement paving quality [1,2]. The most traditional way to detect the pavement paving temperature is to use the plug-in thermometer and temperature measuring gun to measure each process point by point. Due to the small number of measuring points, there are shortcomings of insufficient representativeness of monitoring results and inaccurate regional positioning [3,4].

Based on the above reasons, the application of unmanned aerial vehicle (UAV) infrared detection system is particularly important [5]. Texas Transportation Design and Research Institute successfully applied the infrared sensor and paver to the asphalt pavement construction process [6]. Li Yun analyzed the problems existing in the infrared thermal imager from the technical level, elaborated on the factors affecting the accuracy of the temperature measurement process, and finally, gave relevant solutions [7]. Cheng Shangang developed an intelligent control system for asphalt pavement construction quality based on Internet of Things technology [8]. UAV has more and more applications in intelligent monitoring and measurement in various fields due to their advantages of fast acquisition speed and good security [9,10,11]. Many scholars use UAV equipped with an infrared camera to take infrared images of road surface during asphalt pavement paving and compaction, obtain asphalt pavement compaction information, and timely adjust construction technology to improve the compaction construction quality of asphalt pavement [12,13,14]. However, due to low flight altitude and limited camera focus, the scene field of vision of UAV images is small. Therefore, capturing a relatively complete target area becomes difficult. Multiple images of the same target must be combined by technical means to obtain the complete scene of the required target. The purpose of image mosaic is to combine multiple images with overlapping fields into a panoramic image [15,16,17].

Traditional image mosaic includes feature matching, image matching, beam adjustment, automatic panoramic straightening, gain compensation and multi-band fusion [18]. Region-based stitching algorithm uses greyscale images to achieve registration, which is large in computation and low in registration efficiency [19]. Research on feature-based matching algorithms mainly includes Scale Invariant Feature Transform (SIFT) algorithm and Orientated FAST and Rotated BRIEF (ORB) algorithm. Among them, the ORB algorithm has a short running time and strong robustness. Haichang et al. used Hessian detection operators in the ORB algorithm to deal with the problem of poor scale invariance of ORB. Still, such operators are prone to generate feature point mismatching problems in splicing [20,21,22]. SIFT has the characteristics of rotation invariance, scale invariance, high accuracy of feature point recognition, and is a widely used automatic image mosaic technology [23,24,25,26]. During the paving of asphalt pavement, the temperature of the pavement will decrease with time when the ambient temperature is low. The UAV needs to adjust the position, angle, focal length, and other parameters when shooting each image, which leads to the time difference between the two adjacent infrared images. Further, it leads to the temperature difference in the overlapping area of the two adjacent infrared images. The difference reflected in the infrared image is the color value (including color saturation, color scale, etc.) of the overlapping area of the two adjacent infrared images. The difference of color scales in overlapping areas will cause the splicing error of SIFT.

The low-temperature area in the process of asphalt pavement seriously affects the quality of the pavement. In order to solve the problems of inaccurate recognition of the low-temperature area of the infrared image and the inefficiency of the original SIFT image mosaic method in the process of asphalt pavement paving, this paper provides a method for photographing and splicing the full low-temperature area image of the newly paved asphalt pavement. By converting the infrared image into a binary image, the accuracy of determining the full low-temperature area image of the newly paved asphalt pavement is improved. The splicing reference plate is set to ensure that the splicing is successful when the overlapping area of two adjacent images is small and improve the shooting and splicing efficiency. The proposed method lays the foundation for comprehensive intelligent monitoring and control of asphalt pavement construction quality.

2. Threshold Segmentation Method

The maximum inter-class variance method (Otsu method) [27] divides the image into two parts according to the gray-level characteristics of the image so as to maximize the inter-class variance. The expression of variance is as follows:

$$\sigma (t)=c_0(t){(m_0(t)-m(t))}^2+c_1(t){(m_1(t)-m(t))}^2$$

(1)

where, t represents the gray value of image segmentation and 0 ≤ t ≤ 255, $c_0(t)$ represents the proportion of pixels with gray value between 0 and t in the whole image, $c_1(t)$ represents the proportion of pixels with gray value between t + 1~255 in the whole image, $m_0(t)$ represents the average gray level of pixels whose gray value is between 0 and t, $m_1(t)$ represents the average gray level of the pixel point whose gray level value is between t + 1~255, $m(t)$ represents the total average gray level of the image, $m(t)=(m_0(t)+m_1(t))/2$.

The gray level t when $\sigma (t)$ reaches the maximum value is the best threshold.

3. SIFT Method

SIFT (Scale Invariant Feature Transform) is a feature descriptor. The descriptor has scale invariance and illumination invariance. The scale here can be understood as the image’s blurring degree, which is myopia’s degree. The larger the scale, the less detail. SIFT features want to extract information on all scales. Therefore, scale space is constructed for the image, that is, different smoothing methods are used to check the image for smoothing.

3.1. Determination of Detection Scale Space Extreme Value and Feature Point Principal Direction

Detecting the extreme value of scale space is to search the image position on all scales and identify the points of interest that are invariant to scale and rotation through the Gaussian differential function.

The scale space is defined by the Gaussian kernel as:

$$L(x,y,\sigma )=G(x,y,\sigma )*I(x,y)$$

(2)

where, $I(x,y)$ is the original image, $*$ is the convolution symbol, $L(x,y,\sigma )$ is the scale image under the corresponding scale, (x, y) is the image pixel position, $\sigma$ is the scale space factor, and $G(x,y,\sigma )$ is the Gaussian kernel. The Gaussian kernel can be expressed as:

$$G(x,y,\sigma )=\frac{1}{2\pi {\sigma }^2}{e}^{-(x^2+y^2)/(2{\sigma }^2)}$$

(3)

The image smoothed by the Gaussian kernel is down-sampled to obtain a series of images with different resolutions, namely, image pyramid. SIFT algorithm extracts feature points by establishing the Gaussian difference function. The Gaussian difference image is obtained by subtracting adjacent images of the same scale layer. Then, the obtained difference image is convolved with the original image I(x, y) to obtain the Gaussian difference function. The feature points are obtained by extreme value detection of Gaussian difference function space.

By assigning the direction of the feature points, the image rotation invariance can be achieved. The gradient of the neighboring pixels of the feature points is used to determine its direction parameters, and then the gradient histogram of the image is used to obtain the stable direction of the local structure of the feature points. The gradient histogram statistical method is used to determine the direction of feature points, and the contribution of image pixels in a particular area to the generation of feature point direction is counted with crucial points as the origin.

Collect the gradient and direction distribution characteristics of pixels in the three neighborhood windows of the Gaussian pyramid image where the feature points are located. The magnitude of the gradient can be expressed as:

$$\begin{array}{cc}m(x,y)2& ={(L(x+1,y)-L(x-1,y))}^2\hfill \\ & +{(L(x,y+1)-L(x,y-1))}^2\hfill \end{array}$$

(4)

The direction of the gradient can be expressed as:

$$\theta (x,y)={\tan }^{-1}(\frac{L(x,y+1)-L(x,y-1)}{L(x+1,y)-L(x-1,y)})$$

(5)

3.2. Feature Description and Key Point Matching of Feature Points

Feature points are the key to successful image matching. It is necessary to describe the location, scale, and direction of feature points so as to avoid feature points changing with various changes. The description of feature points includes its own and surrounding pixels contributing to it. This unique description dramatically improves the accuracy of feature point matching.

This paper uses SIFT as a distribution-based descriptor. The specific method to create a descriptor is to treat a region as a 4 × 4. The features in the Gaussian image are associated with this grid. Each bin (angle) of the area is square, and the edge is three times the feature scale. Then, the grid coordinate rotation is equal to the main direction of the desired feature. Calculate the pixel size and gradient direction in the rotation area. The gradient direction rotation is equal to the original direction of the rotation feature. Finally, the SIFT descriptor becomes a vector of 128 components. After the descriptors of all features are created, the matching process is realized by calculating the Euclidean distance of the descriptors in the image. Calculate the Euclidean distance between each feature point descriptor in the reference image and the first and second nearest neighbors in the sensing image and find their ratio. If the value is less than the threshold, it is considered a match point. Set up feature point description subset for template map and observation map, respectively. The feature point descriptors in the two-point set are compared to achieve target recognition. Euclidean distance is used to measure the similarity of feature point descriptors. The smaller the European distance, the higher the similarity. The criterion for determining the success of matching is that the Euclidean distance is less than the set threshold. The Euclidean distance of any two descriptors is expressed as:

$$d(R_i,S_i)=\sqrt{{\displaystyle \sum _{j=1}^{128}(r_{ij}-s_{ij})}}$$

(6)

where, R_i is the descriptor of the feature points in the template graph, $R_i=(r_{i1},r_{i2},\cdots ,r_{i128})$, S_i is the descriptor of feature points in real-time graph, $S_i=(s_{i1},s_{i2},\cdots ,s_{i128})$.

The matching feature point descriptor that meets the requirements shall meet the following conditions:

$$\frac{\mathrm{The} \mathrm{nearest} \mathrm{point} S_i \mathrm{from} R_i \mathrm{in} \mathrm{the} \mathrm{real} - \mathrm{time} \mathrm{graph}}{\mathrm{The} \sec \mathrm{ond} \mathrm{closest} \mathrm{point} R_k \mathrm{from} R_i \mathrm{in} \mathrm{the} \mathrm{real} - \mathrm{time} \mathrm{graph}} <\mathrm{Threshold}$$

SIFT, namely scale invariant feature transformation, is a description used in the field of image processing. This description has scale invariance and can detect key points in the image. It is a local feature description. There are three main processes:

(1)

Extract key points: The characteristics of key points are points that will not disappear due to external factors such as brightness changes, size changes, etc., such as bright spots and dark spots at the edges and corners of the image and in the dark areas. By retrieving all the positions of the image in the scale space, the Gaussian differential function is used to calculate the potential key points.

(2)

Locating key points and determining feature direction: First, the candidate position’s scale and position are determined by using the acceptable fitting model. The stability of location and scale directly affects the selection of key points. Then assign one or more directions to each key. The subsequent operation of image data is realized through the transformation of the direction, scale, and position of key points, thus providing invariance for these transformations.

(3)

Through the feature vectors of each key point, we can find out several pairs of feature points that match each other through pairwise comparison and establish the corresponding relationship between scenes.

The specific process is shown in Figure 1:

3.3. SIFT Infrared Image Mosaic

During the actual asphalt pavement construction, in order to ensure the resolution of the acquired image and obtain a larger area of construction area as much as possible, the unmanned aerial vehicle maintains a fixed height after the on-site flight height adjustment. The unmanned aerial vehicle carries two kinds of camera equipment to take the on-site visible light and infrared images simultaneously, as shown in Figure 2. Therefore, it is necessary to splice the on-site infrared images captured by UAV.

In order to verify the feasibility of using SIFT method to splice on-site infrared images, this paper splices two connected infrared images and confirms the minimum overlapping area of the splicing results using this method. The splicing results are shown in Figure 3.

In the application of this project, after many experiments, if the SIFT method is to be used for image stitching, the overlapping area of two images needs to reach more than 84% before stitching to succeed. More pictures need to be taken to obtain continuous infrared images of the paving site, which increases the workload and reduces work efficiency.

4. Reference Plate

The key to successful image mosaics with SIFT method is that the overlapping area of two images to be mosaiced needs to have enough identical feature points. During the paving process of asphalt pavement, the temperature of asphalt pavement will gradually decrease with time, and the gray value of the overlapping area of the two images to be spliced will be different, so the probability of successful splicing is small. The ultimate purpose of infrared image stitching is to find the low-temperature area of the whole road. However, the road temperature is generally reduced at the same time. Threshold segmentation in the same image can accurately obtain the low-temperature area. By splicing all the binary images containing low-temperature areas, the binary image of the whole road can be obtained, which can intuitively determine the location of the low-temperature area. However, the binary image will reduce the effective information of the image, and the overlapping area needs to be large enough to make the image mosaic successful, which dramatically reduces the efficiency of the mosaic. Sometimes two binary maps to be spliced do not have the same feature points, so they cannot be spliced, as shown in Figure 4. In order to solve this problem, this paper proposes a specific reference plate whose feature points can help to complete the image mosaic successfully. In Figure 5, the black part is an iron plate, the white part in the middle of the iron plate indicates hollowing, and the top angle bisector of the isosceles triangle is parallel to the extension direction of adjacent images. The shape of the reference plate is shown in Figure 5. If the road width is larger, the reference plate of a larger size is selected. Otherwise, the reference plate of a smaller size is selected. For two lanes of pavement, the width of the reference plate is about 20 cm.

5. Example Application

In the asphalt layer paving of an expressway in Shandong, the image Mosaic method proposed in this paper is used, which includes the following steps:

(1)

Place the UAV equipped with an infrared camera at the starting point and adjust the camera lens to make it parallel to the asphalt pavement. As shown in Figure 6.

(2)

Take the infrared image of the newly paved asphalt pavement, as shown in Figure 7. In Figure 7, 1 represents the first infrared image, 2 represents the second infrared image adjacent to the first image, 3 represents the splicing reference plate, 4 represents hollow part, 5 represents overlapping area. Take the first infrared image and the second infrared image.

(3)

Repeat steps (1) and (2) until the infrared image covering all newly paved asphalt pavement is obtained.

(4)

The binary image of the captured infrared image is obtained by using the maximum inter-class variance threshold segmentation method, as shown in Figure 8.

(5)

By splicing adjacent infrared images through SIFT, the reference plate proposed in this paper can provide enough feature points for adjacent image splicing, as shown in Figure 9. Finally, a full low-temperature area image of the newly paved asphalt pavement is obtained, as shown in Figure 10.

The overlap area required for image mosaic using the reference plate is calculated to be 5%, which is 79% lower than the original SIFT method in Section 3.3. In the actual construction process of asphalt pavement paving in order to obtain the continuous infrared image of 1 km of paving, the original SIFT method needs 1043 splicing calculations, and the image splicing method using the reference plate only needs 175 times, which significantly improves the calculation efficiency.

6. Discussion

Compared with the SIFT method without a reference plate, the new method proposed in this paper has several advantages: (1) In order to eliminate the mosaic error caused by the color difference of infrared image, this paper uses the key points based on the shape of the reference plate to match the binary image. This method greatly improves the accuracy and efficiency of image mosaic. (2) In this paper, the reference plate is introduced to ensure the successful splicing of adjacent binary images. The splicing reference plate used is an isosceles triangle. The bisector of the top angle of the isosceles triangle is parallel to the extension direction of the adjacent image. The splicing reference plate is equipped with a hollow part, which can provide more feature points for image splicing and improve the success rate of image splicing. In addition, the circular hollow part makes the splicing reference plate easy to process. As long as the overlapping area contains the circular area of the reference plate, there are enough feature points in the overlapping area to successfully complete the image phenomenon splicing.

The measurement object of the plug-in thermometer and temperature measuring gun is single-point discontinuity, and the representativeness of measurement results is poor. The measurement method of the infrared sensor of the paver is distorted due to the angle problem, and the actual area of the low-temperature area cannot be calculated. The benchmark plate splicing method proposed in this paper has a remarkable effect in the application of asphalt pavement.

The SIFT algorithm-based image mosaic of a UAV infrared system is capable of rapidly acquiring a continuous temperature distribution map of a newly constructed road. By accurately calculating the area of low-temperature regions across the entire road, it is possible to determine their locations. The efficiency of the method proposed in this paper is 5.96 times that of the existing methods. The method is simple and effective, and it is important to realize the intelligent control of asphalt paving quality.

7. Conclusions

Aiming at the problems of discontinuous recognition area of an infrared image used for paving and large overlapping area required for splicing, an image splicing technology based on unmanned aerial vehicle infrared imaging is proposed, which is used to quickly determine the location of the low-temperature area in hot mix asphalt pavement paving construction.

(1)

In this paper, the binary image after threshold segmentation is used for stitching to eliminate the stitching failure caused by different gray values of overlapping regions.

(2)

A reference plate is proposed, which can provide enough feature points and help the front and rear images to be spliced successfully without increasing the overlapping area, significantly improving the splicing efficiency.

(3)

The results of practical application show that the overlapping area required by this method is 79% lower than that of the original SIFT algorithm. It is a very efficient and practical image mosaic method, which provides scientific data guarantee for quality control in the asphalt pavement paving process.