1. Introduction
In Japan, there are more than 700,000 bridges managed by the national and local governments [
1].The percentage of bridges that are 50 years old from date of construction will increase to more than 50 percent in 2029 and it is required by law to inspect these bridges once every five years [
2]. It is difficult to secure the engineers necessary for these inspections; the number of engineers is insufficient for bridge inspection in 20 percent of local governments in Japan. Thus it is desired to realize bridge inspection using a camera mounted on a mobile object such as a drone or a mobile robot in order to solve the future labor shortage and shortage of engineers and improve the efficiency of inspections [
3]. While the elemental technologies and methods for movement and inspection differ depending on the type of robotic technology, the bridge inspection technology introduced by the Ministry of Land, Infrastructure, Transport and Tourism in Japan mainly uses a general-purpose camera with tens of millions of pixels to acquire images (video or still images) [
4]. In order to obtain information necessary for maintenance from the images acquired by the camera, the camera can be mounted on a mobile object such as a drone or a mobile robot. On the other hand, the images captured by the camera have the following problems: The camera tends to be out of focus because the images are taken in a dark place; the performance of the camera is general-purpose, there is a limitation in narrowing down the shutter speed in order to obtain the amount of light necessary for taking pictures because some pictures are taken in dark place; or the acquired images are blurred because images are captured while moving on the mobile platform. The blurred image (hereinafter represented by “degraded image”) makes it difficult to check for cracks or other damages necessary for social infrastructure maintenance [
5]. The motivation of this research is to realize bridge inspection managed by the national and local governments using a general-purpose camera mounted on a mobile object such as a drone or a mobile robot.
The target of our research is that degraded images, captured by a general-purpose camera mounted on a mobile object, can be sharpened by image processing using a general-purpose PC, so that the cracks in the concrete can be easily and accurately confirmed for bridge inspection. For the detection of cracks in concrete structures, there have been studies on sharpening using median filter [
6,
7], Gray–Hough transform [
8] and Gabor filter [
9]. These methods are effective for blurring of focus, but their effects have not been confirmed for “blurring” caused by camera movement. There have been recent studies on bridge inspections using a camera mounted on a mobile robot [
10] or UAV [
11,
12,
13]. In order to obtain the information necessary for bridge inspection from the images acquired by UAVs or mobile robots, it is necessary to perform advanced processing on the images, which is difficult to perform on a general-purpose PC. These prior research examples do not identify the point spread function (hereinafter represented by “PSF”) as the cause of the blur in order to sharpen the blurred image.
On the other hand, we seek the cause of the blur in the PSF and try to sharpen the blur by identifying the model of the PSF. Referring to Kamimura et al. [
14] and Kobayashi et al. [
15], we assume that there are two main degradation factors caused by the camera: PSF of the lens optical system and blur caused by camera shake. PSF of out-of-focus blur (hereinafter represented by “PSF-OOF”), which is a blur caused by PSF of the optical system, is caused by the characteristics of the lens, the focus shift during shooting and the resolution of the image sensor, can be confirmed before or after shooting because it is a constant function unique to each camera and lens. In contrast, blur caused by camera movement during shooting, such as PSF corresponding to motion blur during shooting (hereinafter represented by “PSF-MB”), needs to be estimated on a case-by-case basis since the amount and direction of camera movement varies depending on the time of shooting.
The relationship between an image
g degraded by blurring and an unknown original image
f can be expressed as in Equation (
1) where ∗ is the convolution and
h is PSF.
Ignoring the noise
n, we can obtain
. Theoretically, we can recover
f if we could find
such that
is the inverse of
h for PSF which causes degradation for
f. This kind of restoration for blurring due to degradation is often called sharpening. Deconvolution is the process of estimating the original image
f from a degraded image
g by either knowing
h or implicitly finding the equivalent of
h in an algorithm. There has been a lot of research on sharpening degraded images due to blurring [
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26]. The researches can be roughly classified into two categories: Non-blind deconvolution, in which the PSF, the degradation process, is treated as known, and blind deconvolution, in which the PSF is treated as unknown. In this paper, we treat the problem as one in which the unknown PSF-MB and the known PSF-OOF are mixed before the target image is obtained, as described in
Section 2. As a typical example of research on estimating unknown PSF-MB, Yoneji et al. [
17], estimated the amount and direction of movement by examining the period and direction of stripes in the Fourier transformed image of the degraded image, since
is a sinc function when the target is a motion blur. Oyamada et al. [
18] also worked on estimating the amount and direction of blur movement in the direction of the minimum value from the peak of the cepstrum by using cepstrum analysis. On the other hand, in PSF-MB estimation using Fourier transform and cepstrum analysis, it may be difficult to specify the period and direction of the filtering function
due to the influence of the original image
, which is
after Fourier transform.
PSF-MB estimation using Fourier transform and cepstrum analysis is sometimes affected by the original image , which makes it difficult to specify the period and direction of the filtering function .
So the first thing we noticed was as follows. The blur that occurs in an image due to camera movement is a degradation in which a single point in the original image spreads over a certain range. In general, if the shutter speed is not too slow, the camera movement can be approximated by constant velocity linear motion. The blur can be approximated by a function that spreads linearly in width
w only in the direction
of the optical flow, as shown in Equation (
2) [
27] and this approximates PSF-MB.
Since the cepstrum is the inverse Fourier transform of the logarithmic amplitude spectrum of the degraded image, it can be regarded as an image that emphasizes the frequency components that a certain image has strongly. The spectrum of PSF, which represents linear blur, is a sinc function with periodic zero values [
19]. Given such conditions,
and
are the cepstrum of the original image and PSF-MB, respectively, the cepstrum
of the degraded image can be expressed as
=
+
, ignoring the effect of noise. Thus, if
is known,
can be obtained. However, in this study,
cannot be known, so it may be difficult to accurately determine the minimum value of
, such as when
has a large value around the minimum value, simply by obtaining the cepstrum of the entire degraded image [
20,
21] as described in
Section 2.
Therefore, when multiple candidates for
are obtained in PSF-MB estimation, it is not possible to identify a single motion blur period (amount and direction of movement, hereinafter referred to as “blur amount”) that causes the motion blur, and deconvolution by PSF-MB is performed on the degraded image using each as a candidate for the blur amount. PSF-MB deconvolution must be performed on the degraded image, and the blur amount candidate estimated from the PSF-MB that shows the best sharpening effect must be identified as the blur amount [
22]. For this reason, it is desirable to narrow down (specify one)
candidate in PSF-MB estimation. In this paper, a method to estimate parameters which represent motion blur on the image is mainly discussed, where we assume that there are two main degradation factors caused by the camera, out-of-focus blur and motion blur. Major contribution of this paper is that the parameters can properly be estimated form a sub-image of the object under inspection if the sub-image contains uniform speckled texture. Here, cepstrum of the sub-image is fully utilized. Then, a filter which consists of convolution with the motion blur PSF and out-of-focus blur PSF can be utilized for deconvolution of the blurred image for the sharping with significant effect. Previous research papers have sometimes encountered difficulties in estimating the parameters of motion blur because of the emphasis on generality. Some studies, such as Kamimura et al. [
14] and Molina et al. [
23], have tackled the simultaneous estimation of two unknown “original image
f” and “degraded kernel
h” using Bayesian estimation, but since two unknowns are estimated at the same time, the computation Oliveira et al. [
24] first estimated the direction of the blur amount, and then narrowed down the blur amount candidates by searching for pixel of minimum along that direction. Ji et al. [
25] adopted a method of amplifying the
to make it less sensitive to
. time depends on the initial settings, and the solution may diverge in some cases. These processes require time-consuming, multi-step processing on high-performance PC. On the other hand, our proposed method consists of the following processes: Extracting four small areas including feature points and performing cepstrum analysis; extracting the minimum points in eight neighborhoods using MATLAB functions for the cepstrum analysis results; and narrowing down the minimum points by calculating the average of the four results. The amount and direction of blurring can be narrowed down by simple calculations based on MATLAB functions using a general-purpose PC.
In this paper, the main object is made of concrete, and it is novel in that it is shown that the parameters of motion blur can be well estimated by using the unique speckled pattern on the surface of the object. Referring to Oyamada et al. [
22] and Nohara et al. [
26], we hypothesized that the influence of
can be minimized by extracting proper sub-images in the originally given image and calculating the average of the cepstrum of these images, which in turn would make it possible to narrow down the minimum value candidates of the cepstrum. To verify this hypothesis, we conducted experiments to confirm and examine the following two points using a general-purpose camera [
4] used in actual bridge inspections: 1. Influence on the cepstrum when the isolated point-like texture unique to concrete structures is used as a feature point. and 2. Selection method of multiple images to narrow down the candidate minima of the cepstrum.
By estimating the PSF-OOF and the PSF-MB, and then using the convoluted PSF of these two PSFs, we show that image sharpening with the Richardson-Lucy algorithm can detect almost the entire length of a crack on the surface of a concrete wall, though only half of its actual length could be detected in the degraded image [
28]. In addition to applying cepstrum analysis to the sharpening of images captured by a camera mounted on a moving object by assuming that the amount and direction of movement of a UAV used for bridge inspection is one-dimensional for a short period of time, such as the shutter speed, this study confirmed that it is possible to narrow down the candidates for motion blur in cepstrum analysis, which has been difficult to narrow down the estimation in the past due to the susceptibility of the original image. By making it possible to efficiently sharpen blurred images using a general-purpose camera and a general-purpose PC, it is expected that bridge inspections using UAV will be realized as a solution to the current and future problem of a shortage of engineers in the inspection of more than 700,000 bridges in Japan.
The rest of the paper is organized as follows:
Section 2 presents the Basic outline of the proposed method for the estimation of motion blur.
Section 3 depicts the Experimental procedure and results. We prepared 4 different types of feature point and check the effect of these feature points on a cepstrum analysis and selection method of multiple feature points to narrow down the candidates of parameters of motion blur.
Section 4 provides the Confirmation of the effectiveness of the proposed method. We estimated PSF-MB of the blurred image, captured by general-purpose camera, by using our proposed method and confirmed the sharpening effect of estimated PSF. Conclusion and future work suggestions are depicted in
Section 5.
This paper is an extended version of [
28] and we propose a method for specifying one PSF-MB candidate for each image and this method shows better sharpening result than the method in [
28].