Super-Resolution Reconstruction of an Array Lidar Range Profile

Abstract

Aiming at the problem that the range profile of the current array lidar has a low resolution and contains few target details and little edge information, a super-resolution reconstruction method based on projection onto convex sets (POCS) combining the Lucas–Kanade (LK) optical flow method with a Gaussian pyramid was proposed. Firstly, the reference high-resolution range profile was obtained by the nearest neighbor interpolation of the single low-resolution range profile. Secondly, the LK optical flow method was introduced to achieve the motion estimation of low-resolution image sequences, and the Gaussian pyramid was used to perform multi-scale correction on the estimated vector, effectively improving the accuracy of motion estimation. On the basis of data consistency constraints, gradient constraints were introduced based on the distance value difference between the target edge and the background to enhance the reconstruction ability of the target edge. Finally, the residual between the estimated distance and the actual distance was calculated, and the high-resolution reference range profile was iteratively corrected by using the point spread function according to the residual. Bilinear interpolation, bicubic interpolation, POCS, POCS with adaptive correction threshold, and the proposed method were used to reconstruct the range profile of the datasets and the real datasets. The effectiveness of the proposed method was verified by the range profile reconstruction effect and objective evaluation index. The experimental results show that the index of the proposed method is improved compared to the interpolation method and the POCS method. In the redwood-3dscan dataset experiments, compared to the traditional POCS, the average gradient (AG) of the proposed method is increased by at least 8.04%, and the edge strength (ES) is increased by at least 4.84%. In the real data experiments, compared to the traditional POCS, the AG of the proposed method is increased by at least 5.85%, and the ES is increased by at least 7.01%, which proves that the proposed method can effectively improve the resolution of the reconstructed range map and the quality of the detail edges.

1. Introduction

A three-dimensional range profile can provide key information such as the precise position, geometry, size and shape of the target, which is promising for a wide range of applications in the fields of target detection, urban security, and autonomous driving [1]. The Geiger Mode Avalanche Photon Diode (GM-APD) array-based lidar used in this paper has the advantages of high sensitivity, long detection range, small size, fast response speed, and the ability to simultaneously acquire the distance and intensity information of the target [2,3]. At present, foreign published research has shown that the number of array lidar arrays based on GM-APD has reached 256 × 256, but in the domestic research on its late start, subject to the device limitations, the number of array pixels is only up to 64 × 64 [4]. The subsequent detection task, the distance into the image, contains fewer target information pixels; feature extraction cannot be performed due to the lack of rich target details and edge information, resulting in the acquisition of target details of the edge with limited information about the features, which reduces the accuracy of the detection and recognition [5]. In this paper, a super-resolution reconstruction method is utilized to improve the range profile resolution and enrich the target details and edge information.

The currently existing super-resolution reconstruction methods are mainly categorized into three types: interpolation-based, learning-based, and reconstruction model-based [6,7]. The widely used interpolation-based super-resolution reconstruction techniques are nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. Nearest neighbor interpolation [8] considers the unknown image element to have the same distance value as the nearest neighbor image element, which is based on the principle of taking the distance value of the neighboring point with the shortest Euclidean distance among the four neighboring pixel points around the point to be interpolated as the distance value of that point. The bilinear interpolation [9] principle takes the original range profile with its neighboring four points in the image element distance value of the horizontal and vertical direction of the linear interpolation so as to calculate the distance value of the points to be sampled. Bicubic interpolation [10] takes into account not only the effect of the distances of the four directly neighboring points, but also the effect of the rate of change in the distance values between each neighboring point, which better preserves the details and structures in the range profile. However, the interpolation-based method only relies on its own image information, which makes it difficult to recover detail information effectively.

Learning-based super-resolution reconstruction techniques mainly use deep learning models to learn the mapping relationship between low-resolution images and high-resolution images and generate high-quality reconstruction results. Dong et al. [11,12], for the first time combining a convolutional neural network and super-resolution image reconstruction technology, proposed the SRCNN (super-resolution convolutional neural network) method, through a large amount of convolution of the input low-resolution image feature extraction and its reconstruction effect, and its reconstruction efficiency was higher than that of previous image reconstruction methods. Subsequently, Kim [13] et al. proposed the FSRCNN (fast super-resolution convolutional neural network) method on the basis of the SRCNN method to address the problems of insufficient details in processing images, high network computation, and low network computation rate. Learning-based super-resolution reconstruction techniques often require a large number of high-resolution images to establish a sample library in order to conduct super-resolution reconstruction. Deep learning methods rely on large amounts of labeled data for training and excel in handling complex image details and textures. However, these methods face challenges in scenarios with limited data availability. Additionally, deep learning models often require retraining or fine-tuning to maintain a high performance across different datasets or application contexts, which increases the complexity of their application. The internal mechanisms of deep learning models are also complex, typically regarded as a “black box,” which can pose challenges in applications that demand result transparency and interpretability. The imaging system used in this paper cannot obtain high-resolution images due to the limitations of the detection array, and in the application fields of precision guidance and urban security, it is difficult to ensure that there is a more complete target sample library during the process of super-resolution reconstruction of the target range profile by using a neural network method. Therefore, it is particularly important to propose an image super-resolution reconstruction technique that does not rely on sample learning.

Model-based methods do not require large amounts of training data, which provides a significant advantage in scenarios with limited data availability. Since they are based on well-defined physical and mathematical models, these methods exhibit good generalization capabilities across different scenarios without the need for retraining or fine-tuning for various datasets or application contexts. Additionally, model-based methods typically offer a superior interpretability and transparency, as their mechanisms and results can be understood and analyzed through physical and mathematical principles. Reconstruction model-based methods incorporate the a priori knowledge of the image as constraints into the image super-resolution reconstruction process. This class of methods is able to utilize image a priori knowledge to infer high-resolution image detail information for better preservation of image details, and can combine different a priori knowledge and constraints for image reconstruction with high flexibility and scalability. The projection onto convex sets (POCS) super-resolution reconstruction method belongs to a type of reconstruction modeling method that is able to add the a priori information of the low-resolution image sequences to the reconstruction process and then reconstructs a higher-quality high-resolution image by correcting the projection of the interpolated high-resolution reference image [14]. However, this method has the problems of non-unique solution and low convergence stability, which leads to the presence of artifacts at the edge of the image, and for this reason, some scholars have carried out improvement research. Jian Chen et al. [15] introduced Gaussian gradient map theory for neighborhood consistency information measurement, introduced the gradient information of the image as a convex set constraint, and adaptively selected the iteration step size for pixels with different gradients, which improved the original POCS with a poor retention of edge effects but that lacked the ability to process image detail quality. Fang Yaoxin et al. [16] performed edge detection on the initial high-resolution frame to be reconstructed and then applied the improved point spread function (PSF) to the detected edge pixels, so that the horizontal and vertical coefficients of the PSF corresponding to the pixels at the edges were set to different weights according to the change in the edge slope. The proposed method took into account the specificity of the reconstruction of the edge parts of the image, so that the edge of the reconstructed high-resolution image was well maintained. Wenhao Shao et al. [17] considered the difference between the target region and the background of the image to introduce an adaptive correction threshold and adjusted the size of the correction threshold by utilizing the statistical properties of the edge strength obtained by the Scharr operator, so that the recovered high-resolution sub-millimeter-wave image was close to the real image. Lina Xu [18] et al. improved the original POCS by constructing reference frames using energy successive degradation and motion estimation using the Vandewalle alignment method to determine the exact position of the low-scoring image in the high-scoring grid, which can effectively improve the image clarity and increase detail information.

Fractional calculus and fractional processes, with applications in control systems and image processing, are a hot topic. Many computational fractional intelligence systems and stability analysis and image processing applications have been proposed [19]. R. Liu [20] proposed a new image enhancement algorithm using the combination of rough set and the particle swarm optimization (PSO) algorithm to distinguish the smooth area and edge and texture areas of the image and, according to the results of image segmentation, an adaptive fractional differential filter was used to enhance the image. Xue feng Zhang et al. [21] proposed an image enhancement algorithm based on a rough set and fractional order differentiator. By combining the rough set theory with a Gaussian mixture model, a new image segmentation algorithm with higher immunity was obtained. Hui Yan et al. [22] designed an infrared (IR) and visible (VIS) image fusion algorithm for the injection of the IR objects into the VIS background in a perceptual manner. It ensures that the fused image has both the rich background information of the VIS image and the salient features of the IR image.

Aiming at the problem that the range profiles formed by the current array lidar have low resolution and contain few target details and little edge information, and inspired by the above literature, a POCS super-resolution reconstruction method combining the LK optical flow method and the Gaussian pyramid is proposed in this paper. Considering the distance mutation of the target at the edge background, the gradient constraint is introduced under the data consistency constraint to enrich the edge information of the reconstructed image; finally, projection iteration correction is performed on the reference high-resolution range profile to achieve the improvement in the resolution of the range profile while retaining more detailed edge information of the target. The main contributions of this paper are as follows:

(1)

The LK optical flow method is introduced into the traditional POCS super-resolution reconstruction method, which combines the Gaussian pyramid with multi-scale correction of the estimation vector to improve the motion estimation accuracy and effectively enhance the reconstructed image detail information.

(2)

Considering the sudden change in the target’s distance at the edge background, the gradient constraint is introduced on top of the data consistency constraint, which enhances the reconstruction ability of the target edge.

The rest of this paper is structured as follows: In Section 2, we discuss the details of the proposed method and provide the theoretical analysis. In Section 3, we conduct experimental tests based on the redwood-3dscan dataset and APD array lidar data and present the experimental results. In Section 4, we present the conclusion and discussion of our work.

2. Methods

2.1. POCS Method

POCS is an ensemble-based theoretical method that can flexibly utilize all kinds of prior knowledge, and the relevant prior knowledge can be used as constraints for image reconstruction [23]. The method is an iterative process; the projection operator of the corresponding constraint set projects a point in the solution space to the point closest to the surface of the convex set and repeatedly performs the iterative operation to obtain a solution that converges to the intersection of the convex constraint set [24].

When applying the POCS method for the super-resolution reconstruction of range profiles, it is necessary to establish a link between the original low-resolution range profile sequence and the high-resolution range profile, as shown in Equation (1):

$$d_k\left(x,y\right)={\displaystyle \sum _{\left(i,j\right)}D\left(i,j\right)}\cdot h\left(x,y;i,j\right)+\mu \left(x,y\right)$$

(1)

where $d_k\left(x,y\right)$ is the $k_{th}$ frame of the low-resolution (LR) observation range profile, the $D\left(i,j\right)$ is the original high-resolution (HR) range profile, $h\left(x,y;i,j\right)$ is the point spread function (PSF), and $\mu \left(x,y\right)$ is additive noise.

$C_k\left(i,j\right)$ is the data consistency constraint, which is the basic constraint for recovery reconstruction, defined as shown in Equation (2):

$$C_k\left(i,j\right)=\left\{D\left(i,j\right):\left|r_d\left(x,y\right)\right|\le \delta \right\}$$

(2)

$\delta$ is the correction threshold and $r_d$ indicates the residual difference between the low-resolution distance value and the reference high-resolution estimated distance value, calculated as shown in Equation (3):

$$r_d\left(x,y\right)=d\left(x,y\right)-{\displaystyle \sum _{\left(i,j\right)}D\left(i,j\right)}\cdot h\left(x,y;i,j\right)$$

(3)

For any point $D\left(i,j\right)$ in the reference high-resolution range profile, the projection under constraint $C_k\left(i,j\right)$ is satisfied:

$$D^{\prime }(i,j)=D(i,j)+\left\{\begin{array}{ccc}\frac{r(x,y)-\delta }{{\displaystyle \sum _i{\displaystyle \sum _j{h}^2\left(x,y;i,j\right)}}}·h\left(x,y;i,j\right)& ,& r(x,y)>\delta \hfill \\ 0& ,& \left|r(x,y)\right|\le \delta \hfill \\ \frac{r(x,y)+\delta }{{\displaystyle \sum _i{\displaystyle \sum _j{h}^2\left(x,y;i,j\right)}}}·h\left(x,y;i,j\right)& ,& r(x,y)<-\delta \hfill \end{array}\right.$$

(4)

2.2. Improved POCS Method

The process of POCS super-resolution reconstruction is mainly divided into three parts: constructing the reference frame, motion estimation, and iterative correction by the PSF. Among them, motion estimation is a key step to ensure whether the low-resolution range profile can be accurately mapped to the high-resolution reference range profile, whereas the traditional POCS using the block-matching motion alignment method is unable to provide pixel-level information alignment, which affects the quality of the details of the final reconstructed high-resolution range profile. To improve the accuracy of motion alignment, the Lucas–Kanade optical flow method based on the Gaussian pyramid is introduced for motion estimation of range profile sequences. The accuracy of the motion estimation can be further improved by modeling the motion vectors at different scales through a pyramid structure and performing optical flow calculations using the Lucas–Kanade method, which is iteratively calculated through the pyramid structure [25,26].

2.2.1. Motion Estimation by the Lucas–Kanade Optical Flow Method

The optical flow method estimates the motion state of the target in the image by using the temporal and spatial variations in the distance values in the range profiles of two adjacent frames and their corresponding distance values to obtain the motion vector field. The LK optical flow method is based on the optical flow field and assumes that the optical flow vectors are constant in the region centered on a single point. Set the distance value of point $\left(x,y\right)$ at time $t$ as $I\left(x,y\right)$. This point reaches point $\left(x+dx,y+dy\right)$ at $t+dt$ time, and the corresponding distance value is $I\left(x+dx,y+dy\right)$. According to the image consistency hypothesis, when $dt\to 0$, the distance value of each pixel of the range profile remains unchanged, that is:

$$I\left(x,y,t\right)=I\left(x+dx,y+dy,t+dt\right)$$

(5)

We expanded the right-hand side of Equation (5) by a Taylor series expansion, as in:

$$\frac{\partial I}{\partial x}\frac{dx}{dt}+\frac{\partial I}{\partial y}\frac{dy}{dt}+\frac{\partial I}{\partial t}=I_x\frac{dx}{dt}+I_y\frac{dy}{dt}+I_t=0$$

(6)

Let $u=\frac{dx}{dt}$ and $v=\frac{dy}{dt}$ be expressed in vector form, as in:

$$\nabla I·U+I_t=0$$

(7)

$\nabla I=[I_x,I_y]$ denotes the direction of the gradient from the image and $U={\left[u,v\right]}^T$ indicates the optical flow vector. We used the least square method to solve $U={\left[u,v\right]}^T$.

2.2.2. Gaussian Pyramid Multi-Scale Optical Flow Iteration

A Gaussian pyramid is used for the multi-scale iterative calculation of motion vectors to improve the accuracy of the obtained motion vector. The range profile at the bottom of the pyramid is used to represent the original low-resolution range profile, and the multi-layer range profile with reduced resolution can be obtained by downsampling the original range profile continuously. Then, starting from the top of the pyramid, the optical flow iterative correction is carried out layer by layer downwards, as shown in Figure 1.

Firstly, the optical flow vector $d$ of the low-resolution range profile $L_0$ is first derived as the initial value of the bottom of the pyramid, and the range profile size is set to be downsampled by a factor of $2^{-1}$ at a time. Then, the top layer $L_m$ motion vector is:

$$d^{L_m}=\frac{d}{2^{L_m}}$$

(8)

Secondly, the optical flow value of the top layer $L_m$ is set to $d^{L_m}=d^{L_m}+g^{L_m}$, so that the initial value of the estimated optical flow is $g^{L_m}={\left[0,0\right]}^T$.

Then, the initial estimate for layer $L_m-1$ is:

$$g^{L_m-1}=2\left(d^{L_m}+g^{L_m}\right)$$

(9)

After being solved iteratively layer by layer, the final value of the underlying optical flow $d$ is:

$$d=d^{L_0}+g^{L_0}$$

(10)

2.2.3. Gradient Constraint and Projection Iteration

In order to further improve the edge details of the range profile, gradient constraints are introduced on the basis of the original constraints, and the formula is calculated as follows:

$$\nabla g_d\left(x,y\right)=\left[\frac{\partial d}{\partial x}\left(x,y\right),\frac{\partial d}{\partial y}\left(x,y\right)\right]$$

(11)

$$\nabla g_D\left(i,j\right)=\left[\frac{\partial D}{\partial i}\left(i,j\right),\frac{\partial D}{\partial j}\left(i,j\right)\right]$$

(12)

$$r_g\left(x,y\right)=\nabla g_d\left(x,y\right)-{\displaystyle \sum _{\left(i,j\right)}\nabla g_D\left(i,j\right)}\cdot h\left(x,y;i,j\right)$$

(13)

where $\nabla g_d$ is the local gradient of the low-fraction range profile; $\nabla g_D$ is the reference high-resolution range profile local gradient; and $r_g$ is the gradient-constrained residual, and the final residual is combined as:

$$r\left(x,y\right)=r_g\left(x,y\right)+r_d\left(x,y\right)$$

(14)

The new constraint can be defined as:

$${C^{\prime }}_k\left(i,j\right)=\left\{D\left(i,j\right),\left|r\left(x,y\right)\right|\le \delta \right\}$$

(15)

For any point $D\left(i,j\right)$ on the reference high-resolution range profile, the projection under the constraint ${C^{\prime }}_k\left(i,j\right)$ satisfies:

$$D^{\prime }(i,j)=D(i,j)+\left\{\begin{array}{ccc}\frac{r(x,y)-\delta }{{\displaystyle \sum _i{\displaystyle \sum _j{h}^2\left(x,y;i,j\right)}}}·h\left(x,y;i,j\right)& ,& r(x,y)>\delta \hfill \\ 0& ,& \left|r(x,y)\right|\le \delta \hfill \\ \frac{r(x,y)+\delta }{{\displaystyle \sum _i{\displaystyle \sum _j{h}^2\left(x,y;i,j\right)}}}·h\left(x,y;i,j\right)& ,& r(x,y)<-\delta \hfill \end{array}\right.$$

(16)

To sum up, the improved method firstly adopts the nearest neighbor interpolation that preserves the edge of the range profile to upsample the single low-resolution range profile to construct the high-resolution reference frame. Secondly, the LK optical flow method is introduced to calculate the relative motion vector between low-resolution range profile sequences, and the motion vector is modified with a Gaussian pyramid. Considering the distance difference between the edge and the background, the gradient constraint is introduced under the condition of data consistency constraint. Finally, the reference high-resolution range profile is iteratively modified by projection to improve the resolution of the range profile while retaining more detailed edge information of the target. The overall process of the method is shown in Figure 2.

3. Experimental Results

3.1. Dataset Experiment and Result Analysis

In this paper, the dataset (redwood-3dscan) used in the literature [27] was used for quantitative evaluation, which was a range profile with a resolution of 640 × 480 collected by PrimeSense Carmine camera (Made by PrimeSense, Tel Aviv, Israel). The experiments were verified on a computer with Intel(R) Core (TM) i7-7700HQ and 16 GB memory, and MATLAB R2022a was used as the experimental simulation platform.

In order to verify the reconstruction effect of the super-resolution method, the range profiles of the dataset were firstly downsampled by 1/2, 1/4, and 1/8 to simulate the low-resolution range profiles and then reconstructed by 200%, 400%, and 800%, respectively, and compared to bilinear interpolation, bicubic interpolation, POCS, and the POCS method reproduced in the literature [17]. The average gradient ($AG$) and edge strength ($ES$) were used to objectively evaluate the reconstructed range profile [28].

The $AG$ is the average of the pixel gradient values in the image, and the gradient reflects the rate of change in the pixel intensity in the image. This indicator not only reveals the change trend of the entire image, but also captures the change in the contrast and texture of tiny details, which is closely related to the clarity of the image, and the larger the value, the clearer the image, as in:

$$AG=\frac{1}{M\times N}{\displaystyle \sum _{i=1}^M{\displaystyle \sum _{j=1}^N\sqrt{\frac{{\left(\frac{\partial f}{\partial x}\right)}^2+{\left(\frac{\partial f}{\partial y}\right)}^2}{2}}}}$$

(17)

where $M$ and $N$ represent the number of horizontal and vertical pixels of the range profile, respectively. $\frac{\partial f}{\partial x}$ is the horizontal gradient and $\frac{\partial f}{\partial y}$ is the vertical gradient.

$ES$ is an important index to evaluate edge sharpness and intensity in images. An edge refers to an area where the strength of a pixel changes rapidly, often representing the outline or boundary of the target. The edge intensity is measured by measuring the gradient amplitude of the edge points in the image, and the larger the value, the better the retention effect of the edge information, as in:

$$ES\left(i,j\right)=\sqrt{\nabla xf{\left(i,j\right)}^2+\nabla yf{\left(i,j\right)}^2}$$

(18)

The first-order difference formula of the image along the $x$ and $y$ directions at point $\left(i,j\right)$ is as follows:

$$\left\{\begin{array}{l}\nabla xf\left(i,j\right)=f\left(i,j\right)-f\left(i-1,j\right)\\ \nabla yf\left(i,j\right)=f\left(i,j\right)-f\left(i,j-1\right)\end{array}\right.$$

(19)

For the dataset experiment with the reference high-resolution range profile, we additionally introduced peak signal to noise ratio ($PSNR$) and structural similarity ($SSIM$) [21] to analyze the algorithm performance.

The $PSNR$ measures the quality between the original image and the enhanced image. A higher $PSNR$ value indicates less degradation of the image. It is estimated by calculating the value of mean square error ($MSE$) as:

$$MSE=\frac{1}{MN}{\displaystyle \sum _{m=1}^M{\displaystyle \sum _{n=1}^N{\left(f^{\left(\alpha \right)}\left(m,n\right)-f\left(m,n\right)\right)}^2}}$$

(20)

$$PSNR=10\log \left(\frac{{\left(L-1\right)}^2}{MSE}\right)$$

(21)

The $SSIM$ is an indicator of the similarity between two images, which is defined as:

$$SSIM\left(x,y\right)=\frac{\left(2{\mu }_x{\mu }_{\mathrm{y}}+c_1\right)\left(2{\sigma }_{xy}+c_2\right)}{\left({\mu }_x^2+{\mu }_y^2+c_1\right)\left({\sigma }_x^2+{\sigma }_y^2+c_2\right)}$$

(22)

where ${\mu }_x$ and ${\mu }_y$ are the averages of $x$ and $y$, respectively; ${\sigma }_x^2$ and ${\sigma }_y^2$ are the variances of $x$ and $y$, respectively; ${\sigma }_{xy}$ is the covariance of $x$ and $y$; and $c_1={\left(0.01L\right)}^2$ and $c_2={\left(0.03L\right)}^2$ are constant numbers to stabilize the equation. The range of $SSIM$ is [0, 1]. When the two images are exactly the same, the value of $SSIM$ is equal to 1.

In this experiment, the number of iterations of POCS was set to 5, and the Gaussian model was used to simulate the PSF. The number of layers of the Gaussian image pyramid used in this paper was 3. Figure 3 and Figure 4 are the 400% super-resolution reconstruction effect diagram, and

Table 1. Comparison of the evaluation indicators of the sofa range profiles under different magnifications.

	AG			ES			Time/ms
	$\times 2$	$\times 4$	$\times 8$	$\times 2$	$\times 4$	$\times 8$	$\times 2$	$\times 4$	$\times 8$
Bilinear	0.98	0.94	0.86	10.87	10.50	9.69	13.1	14.2	15.4
Bicubic	1.25	1.04	0.91	12.70	11.10	9.93	13.7	15.1	16.1
POCS	1.42	1.10	0.95	13.98	11.37	10.29	34.3	41.6	49.8
POCS [17]	1.53	1.15	0.99	15.02	11.95	10.76	43.2	48.4	53.3
POCS with LK	1.57	1.17	1.05	14.93	11.89	10.73	45.7	51.2	57.6
POCS with Gradient	1.41	1.09	0.96	15.21	12.03	11.13	41.4	46.2	51.1
Proposed	1.58	1.19	1.05	15.42	12.32	11.21	47.3	53.1	59.7

and

Table 2. Comparison of the PSNR and SSIM of the sofa range profiles under different magnifications.

	PSNR			SSIM
	$\times 2$	$\times 4$	$\times 8$	$\times 2$	$\times 4$	$\times 8$
Bilinear	22.4767	19.6141	17.4205	0.8126	0.7174	0.6017
Bicubic	22.6498	19.754	17.6544	0.7889	0.6707	0.5203
POCS	24.4371	22.2587	18.2147	0.8317	0.7752	0.6932
POCS [17]	25.7124	23.7412	18.8253	0.8617	0.7984	0.7184
POCS with LK	25.5834	23.7368	18.8865	0.8543	0.7897	0.7032
POCS with Gradient	24.9758	22.8907	18.8356	0.8412	0.7834	0.7015
Proposed	26.1342	24.0297	18.8793	0.8832	0.8084	0.7217

are the evaluation index results of different algorithms under different magnifications.

The sofa range profile in the dataset that can effectively distinguish the target edge from the background was selected to verify the effectiveness of the proposed algorithm. At the same time, ablation experiments were added to verify the effectiveness of the introduced LK optical flow method and gradient constraint, where POCS + LK means that the LK optical flow method is introduced to replace the traditional block matching method in POCS motion registration and POCS + Gradient indicates that gradient constraint is introduced in POCS. Figure 3a shows the original range profile; Figure 3b,c represent the range profiles of 400% upsampling bilinear interpolation and bicubic interpolation after 1/4 downsampling, respectively; Figure 3d–h, respectively, represent the 400% upsampling range profile reconstructed by using POCS after 1/4 downsampling and according to POCS, POCS + LK, and POCS + Gradient reproduced in reference [17] and the method proposed in the paper.

As it can be seen from Figure 3, the details at the border of the sofa in Figure 3b are fuzzy and the edge is not sharp enough; Figure 3c is clearer than Figure 3b in terms of image details, but it can be seen from the local enlarged image of Figure 3c that the edge details at the division of sofa and background are not obvious. The reconstruction of range profile details based on the traditional POCS method is better than that of the interpolation method, but the edges are still fuzzy when combined with the local magnification of Figure 3d. In Figure 3e, using the statistical characteristics of the edge intensity obtained by the Scharr operator to adjust the size of the correction threshold, the effect of the reconstructed range profile at the edge is better than POCS, but there are still artifacts on the edge. In Figure 3f, the motion estimation between the low-resolution image sequences is more accurate due to the introduction of the LK optical flow method, and the display effect is clearer than that in Figure 3b–e,g. The edge processing effect needs to be improved, and the target edge contour cannot be effectively distinguished from the background. The gradient constraint is introduced in Figure 3g, and although the edge processing effect is better than that in Figure 3d, the overall detail display effect of the target is worse than that in Figure 3f. In Figure 3h, by introducing the LK optical flow method and gradient constraint, the overall details and edge effects of the target are better than those in Figure 3b–g, which can effectively retain the details and edge information of the target.

In order to further verify the effectiveness of the proposed method in image detail edge processing, the vehicle range profile with abundant target surface details and edge information was selected from the dataset for experimental verification. It can be seen from Figure 4 that the range profile reconstructed based on POCS is better than the overall contour details reconstructed by interpolation. Due to the introduction of the adaptive correction factor in the method proposed in the literature [17], Figure 4e is sharper than Figure 4d at the lights and wheels. Compared to Figure 4f, Figure 4h better highlights the difference between the edge and the background at the edges, such as the wheel, but better shows the overall details of the object. The range profile reconstructed by the method proposed in this paper is sharper in edge and image detail than that proposed in the literature [17] and is the closest to the original range profile.

It can be seen from Table 1 and

Table 3. Comparison of the evaluation indicators of the car range profiles under different magnifications.

	AG			ES			Time/ms
	$\times 2$	$\times 4$	$\times 8$	$\times 2$	$\times 4$	$\times 8$	$\times 2$	$\times 4$	$\times 8$
Bilinear	1.27	1.21	1.01	13.84	13.45	12.07	14.3	15.7	16.9
Bicubic	1.64	1.29	1.08	16.27	16.06	15.17	14.8	16.1	17.4
POCS	2.01	1.36	1.12	19.02	18.79	17.16	36.7	43.2	51.2
POCS [17]	2.27	1.47	1.16	19.58	19.17	17.63	44.8	49.7	55.1
POCS with LK	2.35	1.49	1.20	19.49	18.98	17.54	47.3	52.9	59.4
POCS with Gradient	2.13	1.41	1.14	20.08	19.59	17.97	43.4	47.8	53.8
Proposed	2.38	1.52	1.21	20.35	19.70	18.08	49.6	55.4	61.3

that the POCS super-resolution reconstruction method is superior to the interpolation-based super-resolution reconstruction method in the evaluation indexes of reconstructed images under different magnifications. Our method improves the AG by at least 1.18% and the ES is improved by at least 1.5%, for the sofa range profile reconstruction image quality at different magnifications; for the car range profile at different magnifications, the AG is improved by at least 1.37% and the ES is improved by at least 0.84%. From Table 2 and

Table 4. Comparison of the PSNR and SSIM of the car range profiles under different magnifications.

	PSNR			SSIM
	$\times 2$	$\times 4$	$\times 8$	$\times 2$	$\times 4$	$\times 8$
Bilinear	23.7236	20.4894	18.5695	0.8270	0.7490	0.6563
Bicubic	23.9589	20.6875	18.8796	0.7958	0.6942	0.5709
POCS	25.4371	23.1215	19.3532	0.8621	0.7872	0.7016
POCS [17]	26.7464	24.1632	20.6725	0.8823	0.8057	0.7225
POCS with LK	26.6824	23.9758	20.6346	0.8723	0.7974	0.7183
POCS with Gradient	26.1728	23.5432	20.1325	0.8711	0.7925	0.7097
Proposed	27.1419	25.2231	21.1322	0.8985	0.8154	0.7348

, we can also see that our algorithm outperforms other comparison algorithms in terms of the PSNR and SSIM.

3.2. Real Detecting Data Experiment and Result Analysis

The GM-APD lidar system diagram (a) and the range profile acquisition schematic (b) constructed for the experiment are shown in Figure 5. The system is mainly divided into the laser launching system and echo receiving system; the main process of its detection and imaging principle is to set up a good delay and gate width in the host computer in advance and control the laser to launch laser pulses after a delay time $t_s$. After the GM-APD detector began to work, while the timer began to work, the echo pulse signal triggers the GM-APD; the GM-APD and timer stop working and this time is recorded as $t_d$.

Finally, the distance values corresponding to the triggering of each array element are obtained using the ToF Formula (20):

$$D_i=\frac{t_s+t_d}{2}\cdot c$$

(23)

where $D_i$ is the distance value recorded by the $i_{th}$ probe array element and $c$ is the speed of light.

3.2.1. Indoor Scene Experiment and Result Analysis

For this radar system, we set the laser outgoing re-frequency to $10\mathrm{kHz}$, pulse width to $5\mathrm{ns}$, single-pulse energy to $120\mathsf{\mu }\mathrm{J}$, and the transceiver instantaneous field of view to $0.8°\times 0.8°$.

A $64\times 64$ low-resolution range profile was obtained by detecting a tank model and an armored vehicle model at a distance of 10 m from the lidar in an indoor environment, and 5 consecutively selected from the imaged range profile sequence as a low-resolution image sequence were used as the raw data input.

Since nearest neighbor interpolation is better than bilinear interpolation and bicubic interpolation for image edge detail preservation, this experiment used nearest neighbor interpolation to complete the construction of high-resolution reference frames. Figure 6 and Figure 7 show the effect of a 400% super-resolution reconstructed image, and Table 3 and Table 4 show the results of evaluation indexes of different algorithms under different magnifications.

From Figure 6 and Figure 7, it can be seen that the interpolation-based super-resolution reconstruction method is blurred at the edges of the gun barrel and the rear of the tank model in Figure 6b,c and the front of the armored vehicle and the wheels of the armored vehicle in Figure 6c and Figure 7b, due to the gradual change in the distance value in the processing of the unknown pixels. The traditional POCS method is still better than the interpolation-based method in terms of image details, but due to the large gap between the distance values of the complex areas of the model contour, the edges of the model contour are still fuzzy, and the method proposed in the literature [17] has a clear image, but the edges of Figure 6e and Figure 7e are still slightly artifactual. The method in this paper can effectively improve the fuzzy edges of the model, and the reconstruction effects of Figure 6f and Figure 7f are clear and sharp edges, which can effectively improve the model edge blurring. The reconstruction effect of Figure 6f and Figure 7f is clear and the edges are sharp, which can effectively retain the detailed information of the reconstructed images.

Combining the effect diagrams with the evaluation index results from

Table 5. Comparison of the evaluation indicators of the tank model range profile under different magnifications.

	AG			ES			Time/ms
	$\times 2$	$\times 4$	$\times 8$	$\times 2$	$\times 4$	$\times 8$	$\times 2$	$\times 4$	$\times 8$
Bilinear	2.34	1.27	0.65	23.94	13.81	7.33	4.1	5.3	6.7
Bicubic	3.22	1.69	0.85	30.38	18.08	9.56	4.2	6.4	7.3
POCS	5.06	2.17	1.11	34.96	22.36	11.93	16.2	23.8	31.5
POCS [17]	5.27	2.26	1.16	36.01	23.29	12.47	24.1	29.5	35.2
Proposed	5.46	2.35	1.22	37.41	24.16	12.82	27.2	32.4	39.1

and

Table 6. Comparison of the evaluation indicators of the armored car model range profile under different magnifications.

	AG			ES			Time/ms
	$\times 2$	$\times 4$	$\times 8$	$\times 2$	$\times 4$	$\times 8$	$\times 2$	$\times 4$	$\times 8$
Bilinear	4.28	2.37	1.24	42.60	25.64	13.80	4.9	6.3	7.2
Bicubic	5.98	3.17	1.61	55.26	33.85	18.01	5.2	7.4	8.1
POCS	9.35	3.90	1.97	64.52	38.69	20.85	18.2	25.3	34.2
POCS [17]	10.16	4.34	2.16	68.39	40.54	21.68	25.4	31.2	37.3
Proposed	10.69	4.56	2.24	71.48	42.21	22.52	29.5	35.3	42.6

, it can be seen that the details and edge information contained in the range profile of the armored vehicle model are significantly higher than those of the tank model range profile, which is due to the fact that the GM-APD is a single-photon probabilistic detector. The surface contour of the armored vehicle model is more complex than the surface of the tank model, the number of photons returned by the armored vehicle is lower than that of the tank model, and the number of gaps between the wheels is large, resulting in significantly more voids in the final range profile than in the tank model. The number of photons returned from the armored car is lower than the number of photons from the tank model, and there are considerable gaps between the wheels, which leads to the fact that there are more holes in the final range profile than in the tank model. As a result, the edge information of the range profile of the armored car model is significantly larger than that of the tank model.

3.2.2. Outdoor Scene Experiment and Result Analysis

In order to further verify the reliability of the method, under the settings of the same radar system experimental parameters, the detection imaging of the outdoor scene at a distance of 280 m from the lidar was carried out, and the detection targets included pedestrians riding bicycles, SUV cars, and sedans, in which the SUV cars were in the state of being obscured. Figure 8 shows the 400% super-resolution imaging effect.

From Figure 8b,c, it can be seen that the interpolation-based super-resolution range profile of the target edge and contour details are fuzzy, and when the target and the target or target and background distance values are similar, for example, pedestrians riding bicycles and in close proximity of the side car, the pedestrian is easily interfered with by the bicycle and the car, resulting in the reduction in the accuracy of the extracted feature information. Figure 8f clearly shows the target contours of pedestrians, bicycles, sedans, and SUVs, and it can be seen in

Table 7. Comparison of the evaluation indicators of the outdoor scene range profile under different magnifications.

	AG			ES			Time/ms
	$\times 2$	$\times 4$	$\times 8$	$\times 2$	$\times 4$	$\times 8$	$\times 2$	$\times 4$	$\times 8$
Bilinear	0.22	0.12	0.06	2.35	1.35	0.68	4.4	6.1	7.8
Bicubic	0.28	0.14	0.07	2.74	1.58	0.83	5.8	7.9	8.7
POCS	7.13	4.66	3.76	69.06	44.57	29.91	17.8	24.7	33.5
POCS [17]	7.31	4.78	3.85	74.29	47.38	31.15	24.6	30.4	36.1
Proposed	7.59	5.03	3.98	78.84	51.11	33.08	28.3	34.1	41.3

that the proposed method improves the AG by at least 4.15% and the ES by at least 6.73% compared to the above-compared methods.

As the super-resolution of the range profile increases, the retention of both the detail and edge information of the image by the super-resolution method decreases, but the POCS-based super-resolution reconstruction method still outperforms the interpolation method for the retention of the detail and edge information of the image at the same magnification.

By analyzing the effect of the reconstructed image and objective evaluation indexes, it is proved that the method in this paper has a better edge detail retention ability than bilinear interpolation, bicubic interpolation, and the POCS super-resolution reconstruction methods and meets the demand of the super-resolution reconstruction of range profiles.

4. Conclusions

A POCS super-resolution reconstruction method combining the LK optical flow method and the Gaussian pyramid is proposed in this paper. The method solves the problems of the low resolution of range profiles formed by array lidar and the low retention of target details and edge information by the traditional super-resolution methods.

We compared the method presented in this paper with bilinear interpolation, bicubic interpolation, POCS, and POCS with adaptive correction thresholding using the dataset and the actual data taken from the laboratory-built GM-APD lidar system.

The experimental results show that the proposed method outperforms the other methods in terms of the AG and ES under different super-resolution conditions. In the experiments on the redwood-3dscan dataset, the method proposed in this paper improves the AG by at least 10.12% and the ES by at least 7.02% over the conventional POCS; in the experiments with real mining data, the method proposed in this paper shows at least 9.11% improvement in the AG and at least 9.03% improvement in the ES over the conventional POCS.

It can be seen that the super-resolved range profile can also clearly display the detailed contours of different targets when the distance between the targets is close, which proves that the proposed method can effectively retain the target details and edge information while improving the resolution of the range profile.