A Novel Method for the Deblurring of Photogrammetric Images Using Conditional Generative Adversarial Networks

Abstract

The visual data acquisition from small unmanned aerial vehicles (UAVs) may encounter a situation in which blur appears on the images. Image blurring caused by camera motion during exposure significantly impacts the images interpretation quality and consequently the quality of photogrammetric products. On blurred images, it is difficult to visually locate ground control points, and the number of identified feature points decreases rapidly together with an increasing blur kernel. The nature of blur can be non-uniform, which makes it hard to forecast for traditional deblurring methods. Due to the above, the author of this publication concluded that the neural methods developed in recent years were able to eliminate blur on UAV images with an unpredictable or highly variable blur nature. In this research, a new, rapid method based on generative adversarial networks (GANs) was applied for deblurring. A data set for neural network training was developed based on real aerial images collected over the last few years. More than 20 full sets of photogrammetric products were developed, including point clouds, orthoimages and digital surface models. The sets were generated from both blurred and deblurred images using the presented method. The results presented in the publication show that the method for improving blurred photo quality significantly contributed to an improvement in the general quality of typical photogrammetric products. The geometric accuracy of the products generated from deblurred photos was maintained despite the rising blur kernel. The quality of textures and input photos was increased. This research proves that the developed method based on neural networks can be used for deblur, even in highly blurred images, and it significantly increases the final geometric quality of the photogrammetric products. In practical cases, it will be possible to implement an additional feature in the photogrammetric software, which will eliminate unwanted blur and allow one to use almost all blurred images in the modelling process.

1. Introduction

Unmanned aerial vehicle (UAV) photogrammetry and remote sensing have become very popular in recent years. UAVs as platforms for support research equipment enable reaching various regions and terrains, often inaccessible to traditional manned solutions. Their trajectory can be remotely controlled by humans or programmed and implemented automatically. The development and operation of commercial unmanned aerial vehicles is rapid and has become very simple owing to their commercialization. Within a short period, scientists and engineers from all over the world noticed these advantages and widely began using these devices to transport research equipment. As a result, engineering measurements and tests started to be conducted in new places with unprecedented frequency.

Commercial UAVs are the machines most commonly used for low-altitude photogrammetry. Their maximum take-off mass (MTOM) is low, typically not exceeding 25 kg, as is their capacity for carrying additional load. This enforces the need to reduce the weight of all components carried by such a vehicle. Miniaturization involves, among others, global navigation satellite system (GNSS) receivers, inertial navigation systems (INSs) and optoelectronic sensors (visible light, thermal imaging, and multispectral cameras), often making these devices less sophisticated and accurate. Unmanned platforms most commonly use single-frequency GNSS receivers, with a positioning accuracy of up to several meters [1,2,3]. Only recently have the engineers started fitting UAVs also with GNSS RTK (Real Time Kinematic) and PPK (Post-Processed kinematic) precise satellite navigation receivers, where the achieved accuracies are already at a level of 0.10 m [4,5,6,7]. Inertial sensors based on micro electromechanical systems (MEMSs) are used in relation to determining the UAV angular orientation values. The accuracy of determining angular values in inertial units of this type varies around 1° for transverse and longitudinal inclination angles, and 2° for the torsional angle [8]. Optoelectronic cameras carried by UAVs are small structures based on CMOS (Complementary Metal-Oxide-Semiconductor) sensors, CCDs (charge-coupled devices) or uncooled VOx (Vanadium Oxide) microbolometers, equipped with prime lens with a conical field of view. These are mainly non-metric cameras, where the internal orientation elements are not stable [9].

The photogrammetric process involves numerous factors impacting the quality of the end result [10,11,12,13,14,15,16,17,18,19]. In the course of research described herein, they were divided into three groups:

• Procedural—All elements of the image and additional data collection process, which originate from the adopted execution method and its correctness. This group contains the parameters of the applied flight plan (altitude, coverage, selected flight path and camera position), GNSS measurement accuracy (RTK), selected time of day, scenery illumination quality, etc.;
• Technical—Factors impacting the study quality, arising from the technical parameters of the devices used to collect data. This group contains the parameters and technical capabilities of the cameras, lens, stabilization systems, satellite navigation system receivers, etc.;
• Numerical—Factors originating from the methods and algorithms applied for digital data processing.

Maintaining the compilation quality requires good knowledge in the field of the aforementioned elements and their skillful balancing, as well as matching them with encountered measurement condition and faced requirements. If one of these groups remains unchanged (e.g., the technical one), the assumed quality requirements can be maintained by adjusting the parameters of other groups. In other words, with available constant technical resources, it is possible to adjust the numerical methods and procedures so as to achieve the objective. For example, if one has a given UAV with a specific camera and lens (fixed element), it is possible to match the flight altitude to achieve the required ground sampling distance (GSD). Otherwise, when it is impossible to change the flight altitude, which means that the procedural factor is constant, one can use another camera (lens and spatial resolution) or, alternatively, apply numerical methods in order to achieve the required GSD.

With the available constant technical element, the parameters of the procedural and numerical groups are adapted in order to reach the accuracy-related requirements of the outcome. The quality of the study can be significantly improved by properly selecting these elements. Some of them are determined analytically, at the task execution planning stage, e.g., flight altitude, speed, shutter speed, etc., while others are the result of agreed principles and adopted good practices. They primarily result from conducted research and gained experience. The research [20] presents the impact of significant coverage on the quality of point clouds generated within the photogrammetric process. The paper [21] discusses the results of experiments involving various flight plans and assesses their impact on the geometric quality of the developed models. The case study [22] reviews the results of a study originating from air and ground image data. The research [23] combines point clouds from terrestrial laser scanning (TLS) and ground image data. Similarly, the experiment [24] utilized the technique of fusing data from aerial laser scanning and optical sensors mounted on a single unmanned aerial platform. The authors of the publication [25] maximize aircraft performance, and in [26] put emphasis on flight correctness and proper planning. Several interesting scenarios and recommendations in terms of spatial modelling techniques were thoroughly discussed in [27].

It should be noted that an appropriate procedure for acquiring photos and its parameters (flight plan and its parameters) must be selected prior to their physical execution. However, there may be cases where the photos had already been taken, do not meet the assumed requirements and it is impossible to repeat the measurement. In general, it can be assumed that two elements impacting the quality of a photogrammetric study are constant, namely procedural, which is the inability to repeat a procedure with other parameters, and technical, which is the inability to change the technical parameters of the used measuring equipment. In such cases, in order to achieve a required study quality, one can apply specific numerical methods. Therefore, the result quality can be improved by using numerical methods. The author of [28] managed to significantly reduce ground sample distance and increased numbers of points in the dense point cloud using a fixed camera and flight altitude. The papers [13,16,18] review a method for improving the radiometric quality of photos. In [29], the researchers presented a method to improve the camera’s internal orientation elements and their impact on reconstruction quality. In the work [30], the results of improving a digital terrain model using the Monte Carlo method were reviewed. The authors of [31] discuss a thorough impact analysis regarding various feature detectors as well as reconstruction accuracy, whereas the publication [32] presents a new algorithm for the integration of dense image matching results from multiple stereo pairs.

The data acquisition process may encounter a situation in which blur appears on the images. Deblurring in aviation cameras is achieved by a forward motion compensation system (FMC), and blurring is due to aircraft or satellite forward motion [33]. For this purpose, analogue cameras have a mobile image plane, which mechanically compensates for the camera’s forward motion [34]. This compensation is known as time delayed integration (TDI) [35,36,37,38,39]. Blur compensation systems require precise synchronization with the vehicle’s navigation systems. Such solutions are not used in small and inexpensive cameras on commercial UAVs.

Image blurring caused by camera motion during exposure significantly impacts the photogrammetric process quality [40]. On blurred images it is difficult to visually locate ground control points, and the number of identified feature points decreases rapidly together with increasing blur kernel [26,41]. Hence, improving a blurred image can increase the number of detected features and, consequently, the geometric quality of generated point clouds and spatial models.

The authors of [42] divided deblurring methods into two groups, namely blind [43,44] and non-blind deconvolutions [45,46]. Both solutions are aimed at eliminating blur; however, the non-blind convolutions utilize information on the nature of the blur within an image, hence matching its elimination method. Blind methods do not have such knowledge. A different division is suggested by the authors of [47], who put them into seven groups, which are edge sharpness analysis, depth of field, blind de-convolution, Bayes discriminant function, non-reference block, lowest directional frequency energy and wavelet-based histogram. Apart from the aforementioned groups, there are also sensor-assisted blur detection [40,42,48,49] and the latest neural methods [50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65].

The methods for eliminating blur formed on aviation images assume that blur is of a linear nature, which is the case most often. This assumption is also adopted by FMC systems. When obtaining data using a UAV, the flight can be performed according to highly varied plans, at varying altitudes and distances to the object, and in different lighting conditions [66]. For this reason, the nature of blur can be non-uniform, which makes it hard to forecast [67,68]. Due to the above, the author of this publication concluded that the neural methods developed in recent years were able to eliminate blur on UAV images with an unpredictable or highly variable blur nature.

This paper presents the following new solutions in the field of photogrammetry and remote sensing:

• A new, rapid method based on generative adversarial networks (GANs) was applied for deblurring;
• A GAN network model was selected and applied, which can be implemented in UAVs and activated in real-time (highly practical significance of the solution);
• A set of neural network training data was developed based on real aerial images, collected by the author of the study over the last few years;
• More than 20 full sets of photogrammetric products were generated, including point clouds, orthoimages and the digital surface models, from blurred photos and photos subjected to the presented method;
• Techniques for improving photogrammetric photo quality were grouped and a tri-element model of a photogrammetric survey quality was presented.

The publication is divided into four sections. The Section 1 is the Introduction, which presents motivation behind the research. The Section 2, Materials and Methods, describes tools and methods used to process the data. Section 3 presents and discusses the obtained results. The paper ends with the Conclusions, where the most important aspects of the research are discussed and summarized.

2. Materials and Methods

2.1. Deblurring Method

A heterogeneous image blur model can be expressed as the following equation:

$$I_B=k\left(M\right)*I_S+N,$$

(1)

where $I_B$ represents an image with blur, $k \left(M\right)$ are unknown blur kernels determined by a motion field M, $I_S$ represents a sharp latent image, and N is the additive noise. The * symbol means convolution.

Early deblurring methods required the knowledge of the k(M) function, which means they can be classified as non-blind deconvolution methods. In order to perform deconvolution and obtain an image $I_S$, they were primarily based on Lucy–Richardson algorithms [69] and Wiener [70] or Tikhonov [71]. In the case of images obtained by commercial UAVs, we assume that the blur function is unknown. As already mentioned, inexpensive non-metric cameras are not fitted with any technical devices for blur kernel estimation. In such a case, it can be assumed that the application blind deblurring methods, which estimate the blur kernel k(M) can lead to obtaining a latent image $I_S$. Unfortunately, these methods are intended for images with uniform blur, generated as a result of uniform camera motion relative to the object or background [72,73,74,75]. In the case of UAV images, assuming that the k(M) function is the same for the entire image surface is a great simplification and can lead to erroneous results. Some authors have identified this issue and are working on numerical methods based on a non-uniform blur model. The publications [76,77] present a non-uniform blind deblurring model based on a parametric geometric model of the blur process, which additionally takes camera rotation into account. The paper [78] assumes that blur is induced by a complex spatial motion of the camera. Therefore, it can be assumed that the estimation of the k(M) function for UAV images, taking into account all possible flight plans and cases is a very difficult task.

A rapid development of neural methods has been observed in recent years. Many of these have been successfully applied in photogrammetry and remote sensing [79]. Convolutional neural networks (CNNs) have been successfully used for eliminating blur [59,80,81,82,83]. In such solutions, the networks are trained using image pairs. Such pairs form sharp images with synthetically applied blur. Hence, CNNs are trained to estimate the blur kernel, which is then used for deconvolution.

The aforementioned methods estimate the blur kernel or assume its certain nature, followed by deconvolution. Another approach includes generating synthetic images without blur, based on their blurred counterparts. The task of converting one image type to another has only been made possible recently and can be implemented owing to generative adversarial networks (GANs) [84]. For example, a network generates an image based on a certain tag fed at the input [85]. Thus, in the case of a blur estimation problem, a GAN must generate an artificial image without blur, based on input data with blur. Therefore, the GAN training process shall involve supplying two training image groups—a sharp image and a corresponding blurred image.

Generative adversarial networks (GANs) [86] is one of the methods for generative modelling, based on differentiable generating networks. Generative adversarial networks are based on a theoretical game scenario, in which a generating network G must compete with a foe D. A generating network directly generates samples $x=g\left(z;{\theta }^{\left(g\right)}\right)$. Its adversary, a discriminator network, attempts to differentiate between samples extracted from training data and samples extracted from a generator. A discriminator network emits the probability values based on $d\left(x;{\theta }^{\left(d\right)}\right)$, which means the probability that x is a real training example, and not a false sample extracted from a model.

Training formulation in generative adversarial networks involves a zero-sum game, where the $v\left({\theta }^{\left(g\right)},{\theta }^{\left(d\right)}\right)$ determines the reward for the discriminator network. The generator receives $-v\left({\theta }^{\left(g\right)},{\theta }^{\left(d\right)}\right)$ as its reward. In the course of learning, each player attempts to maximize its rewards, so that in the convergence [87]:

$$g*=\arg \underset{\mathrm{g}}{\min }\underset{\mathrm{d}}{\max }v\left(g,d\right)$$

(2)

The default choice for v is:

$$v\left({\theta }^{\left(g\right)},{\theta }^{\left(d\right)}\right)={\mathbb{E}}_{x~p_{data}}\log d\left(x\right)+{\mathbb{E}}_{x~p_{\mathrm{model}}}\log \left(1-d\left(x\right)\right)$$

(3)

The analysis of deblurring methods based on GANs [52,53,55,56,59,61] showed that they coped with heterogeneous blur well. These methods were developed in order to improve the quality of videos recorded by portable video devices, such as sports cameras and smartphones. For this reason, engineers started to optimize the developed algorithms in terms of operating speed, so that it was possible for them to operate in real-time or near-real time [64]. The DeblurGAN-v2 [64] method was chosen out of the aforementioned ones since its presented network model performed the operations the fastest and it enabled implementation directly on UAVs. An example of such a UAV with an on-board computer enabling the implementation of advanced real-time operations on an image is shown in [88]. DeblurGAN-v2 conducts operations up to 100 time faster than its closest rivals, yet maintaining quality close to other leading methods. This method allowed to eliminate blur on a video stream in real-time. A standard HD (High Definition) video stream generates approximately 30 1280 × 720 pixel images per second. In the course of the analyzed study [64], DeblurGAN-v2 eliminated the blur from a set of test data [59] recorded with an HD GoPro camera (GoPro, Inc., San Mateo, USA), in real-time. These results enable a conclusion that this method will be able to eliminate blur for single 3000 × 4000 pixel images in-flight, between successive photos in a series, i.e., every approx. 2 s.

For the purposes of the study, a network model (Figure 1) was implemented in a PyTorch environment [89]. The network training was conducted using a Tryton supercomputer (computing power of 1.5 PFLOPS, 1600 computing nodes, 218 TB RAM) of the Academic Computer Centre in Gdańsk. Next, the model was launched on a computer with a GeForce GTX 1060 graphics processor (GPU) and an CPU Intel(R) Core i7-7700K 4.20GHz processor.

In order to verify the thesis of this paper, a test schedule and computation process were developed. The computation process in graphic form, together with used tools is shown in Figure 2. The test data were acquired using a commercial UAV. Models generated based on the data were adopted as reference models. The next stage is adding blur to the photos. A total of 10 sets of photos with a varying, increasing blur kernel were generated, and then used to generate 10 photogrammetric models. Next, all sets with blur were subject to the DeblurGANv2 method, which consequently enabled us to generate 10 new sets of image data, once again used to develop 10 photogrammetric models, using the same processing settings, as in previous cases. The details of these operations are shown in the further section of this paper.

2.2. Data Acquisition

The photogrammetric flight was conducted using a DJI Mavic Pro (Shenzhen DJI Sciences and Technologies Ltd., China) UAV. Such an unmanned aerial vehicle is a representative of commercial aerial vehicles, designed and intended primarily for recreational flying and for amateur movie-makers. The versatility and reliability of these devices was quickly appreciated by the photogrammetric community. Their popularity results mainly from the operational simplicity and intuitive software [28,66,90].

The flight was programmed and performed as per a single grid plan [91] over a landfill being reclaimed. The progress of the work in this area is periodically monitored using photogrammetry and terrestrial laser scanning. The dimensions of the test area are 240 × 505 m, with the flight conducted at an altitude of 80 m AGL, with a longitudinal and transverse overlap of 75%. A total of 224 photos were taken and marked with an actual UAV position. The UAV position is saved in the EXIF metadata (Exchangeable Image File Format). In addition, the photogrammetric network was developed. Six ground control points (GCPs) were evenly arranged throughout the entire study area, and their position was measured with the GNSS RTK accurate satellite positioning method. GCP position was determined relative to the PL-2000 Polish flat coordinate system and their altitude relative to the quasigeoid. Commercial Agisoft Metashape ver. 1.6.2 (Agisoft LLC, Russia) software was used to process the image data.

In total, 224 images were acquired in-flight, which were then used, without any modifications, to generate a reference model. The results for the reference model, read from a report generated by the software, are shown in

Table 1. Accuracy data of the developed reference model.

Flight Altitude	Ground Resolution	Tie Points	Projections	Reprojection Error
74.5 m	2.45 cm/pix	416,194	1,515,694	1.09 pix
Camera locations and error estimates
X error (m)	Y error (m)	Z error (m)	XY error (m)	Total error (m)
1.01606	2.70717	112.933	2.89156	112.97
GCP locations and error estimates
X error (cm)	Y error (cm)	Z error (cm)	XY error (cm)	Total (cm)
1.35198	0.928474	0.14282	1.6401	1.6463

and the visualization are presented in Figure 3.

2.3. Blurred Data Set

A data set was developed for the GAN model training. The data set contains images acquired using the UAV. All images used were collected during many previous studies conducted by the author. The images mainly represent different types of terrain covered with vegetation and city areas, which were designed to develop typical photogrammetric models. The data set consists of two groups, namely images used for training and images used for trained model verification. These are different photos, which are not present in any of the two groups. The images used to train the neural network cannot be used to validate the trained model. Image pairs, sharp and with added random blur, were generated in each group.

The blur algorithm was implemented in the Python environment, using an OpenCV library [92], according to the following relationship:

$$\mathrm{h}\left[m,n\right]=\left(f*g\right)\left[m,n\right]={{\displaystyle \sum }}_{j,kϵK}f\left[j,k\right]g\left[m-j,n-k\right],$$

(4)

where: $h\left[m,n\right]$ represents a resulting image with a size of $m\times n$, $f\left[j,k\right]$ represents a 2D discrete convolution function, and the source image with the dimensions of $m\times n$ is described in the form of $g\left[m,n\right]$. It was assumed that $K=\left\{5,10,15,\dots ,50\right\}$, whereas the values of a $f\left[j,k\right]$ 2D function for line $k=\left(K-1\right)/2$ take the values of $1/K^2$, with the remaining ones taking 0.

The aforementioned relationships were used to generate both training data, as well as research data. Image data acquired during a reference flight were blurred. A total of 10 sets of research data, with a varying, increasing blur, were generated. The blur was changed by changing the blur kernel K size and adopting a uniform K for each research set.

Next, the same process of generating typical low-ceiling photogrammetry products was conducted using the photogrammetric software. This process followed the same procedure and processing settings as the ones applied for generating reference products. The result was 10 products generated from images with blur.

Table 2. Reported survey data for blurred dataset.

Blur Kernel	Flying Altitude (Reported) (m)	Ground Resolution (cm/pix)	Tie Points	Projections	Reprojection Error (pix)
0	74.5	2.45	416,194	1,515,694	1.09
5	75.6	2.69	459,985	1,703,465	1.01
10	75.9	2.69	468,013	1,762,049	0.969
15	157	2.44	473,566	1,781,602	6.95
20	129	2.49	451,838	1,648,334	6.16
25	79.3	2.64	424,529	1,503,363	1.12
30	70.6	2.72	402,741	1,380,426	1.17
35	10 000	724	378,829	1,255,205	1.24
40	79.6	2.65	363,984	1,169,618	1.18
45	79.6	2.65	346,471	1,089,462	1.19
50	81.1	2.7	335,285	1,030,242	2.99

presents basic data of the surveys with data subject to blurring. It should be noted that a real flight altitude (75 m) was fixed, and the one shown in the table was calculated by the software.

Table 3. Average camera location error. X—Easting. Y—Northing. Z—Altitude.

Blur Kernel	X Error (m)	Y Error (m)	Z Error (m)	XY Error (m)	Total Error (m)
0	1.01606	2.70717	112.933	2.89156	112.97
5	1.05511	2.6617	113.921	2.8632	113.957
10	1.04322	2.65756	114.203	2.85498	114.238
15	25.839	31.9469	204.859	41.0884	208.939
20	19.5152	25.4927	175.837	32.1048	178.744
25	1.30977	2.6244	118.957	2.93308	118.993
30	0.945613	2.91196	108.394	3.06164	108.437
35	0.988093	3.01637	106.259	3.17409	106.306
40	1.14262	2.60115	118.828	2.84105	118.862
45	1.03106	3.17542	118.685	3.33862	118.732
50	10.9788	7.90267	120.892	13.5273	121.646

shows a mean camera location error, and

Table 4. Control points root mean square error (RMSE). X—Easting. Y—Northing. Z—Altitude.

Blur Kernel	GCP Count	X Error (cm)	Y Error (cm)	Z Error (cm)	XY Error (cm)	Total (cm)
0	6	1.35198	0.928474	0.14282	1.6401	1.6463
5	6	1.62547	1.17136	0.17614	2.00356	2.01128
10	6	1.82308	1.1612	0.213706	2.16148	2.17202
15	6	27.0598	35.9852	11.3785	45.0241	46.4396
20	6	22.1977	33.1699	13.7652	39.9122	42.2192
25	4	1.57333	0.498542	0.188985	1.65043	1.66121
30	5	0.961179	0.830387	0.414181	1.2702	1.33602
35	4	0.797182	0.634285	0.278878	1.01873	1.05621
40	3	1.45667	0.158248	0.237339	1.46525	1.48434
45	3	1.19554	0.308544	0.237474	1.23471	1.25734
50	3	2.70446	3.85804	0.234899	4.71154	4.71739

shows the root mean square error (RMSE) calculated for control point locations. In the course of the photo processing process, the control points were manually indicated by the operator for each data set. The points were marked until the moment, when it was impossible to recognize them in the photos and possible to mark their center or the probable GCP center. The fact that it was possible to develop models from all data with blur, and that the software selected for this task completed the process without significant disturbance is noteworthy.

The measurement performed by an on-board GPS records the position and altitude relative to the WGS84 ellipsoid. The GCP position was determined relative to the Polish PL-2000 flat coordinate system and the altitude relative to PL-EVRF2007-NH quasigeoid. As presented in Table 3, the error values for the sharp data set (kernel value 0) in the horizontal plane (x,y) are within the range typical for GPS receivers used in commercial UAVs. The absolute error in the vertical plane (z) is significantly greater and results directly from using different altitude reference systems. Total error for camera location correlates with total RMSE presented in Table 4. The GPS remains unchanged during the flight and it should be assumed that the position accuracy is relatively constant throughout the flight. The total camera location error relates to the positions of the measured control points, the accuracy of which decreases significantly above kernel value 20, and should not be considered as valid.

The control points were manually marked by the operator and up to blur kernel 20 it was possible to mark all 6 points. In a normal case, in terms of practical work, a number of 6 GCPs is the lowest reasonable number to achieve satisfactory results. Up to kernel 20, the total RMSE is growing with maximum at a blur kernel equal to 20. For a blur kernel greater than 20, some of the points were not sharp and visible enough and were not considered as correct ones and not marked in the software. Consequently, errors were calculated for a smaller number of GCPs, and total RMSE values for a blur kernel > 20 is lower that values for all 6 points marked on kernel 20. Generally, using 4 or 3 GPSs may induce errors for the final results and is not recommended in the practical cases. Despite the fact that total RMSE for a blur kernel > 20 could be satisfactory, the number of marked GCPs is too low and should be treated as a non-representative GCP’s error result and not used in the practical cases. Practically, results represented in total RMSE are satisfactory for a blur kernel up to 10, where the operator was able to mark all GCPs and total RMSE at around 2 cm.

2.4. Deblurring with GAN

All sets with applied blur were individually subjected to the deblurring process. For this purpose, images with blur were successively provided as input data for a trained generative adversarial network model. The resulting images for each set were used to generate ten further sets of photogrammetric products. This process was similar to the one involving reference models, with the same processing software settings being used. Figure 4 shows a visual comparison of the results for the DebluGANv2 algorithm for GCP No. 1. The images show, respectively, a reference photo (sharp), photos with blur for kernel 5, 15, 25 and 50, and the corresponding images with blur reduction.

A visual analysis of the images indicates that this image significantly reduces blur to a level of K = 25. This method is particularly effective with sharp and clear edges, lines generated by anthropogenic structures, such as buildings, borders between various ground surfaces, cars, sharp shadows, etc. This phenomenon enables a visual improvement in the recognizability of ground control points and other objects of artificial (anthropogenic) origin (Figure 5). Control point edges in images with blur are clearly blurred, which hinders its proper indication in a photogrammetric program. In practice, it becomes very difficult with a blur kernel above 15. A significant improvement in the sharpness of the control point edge can be noticed in photos subjected to deblurring. When it comes to the control point, this phenomenon improves its recognizability in a terrain and enables its more accurate indication in a photogrammetric program.

The assessment of image qualities and the evaluation of the DeblurGANv2 method in comparison with the blurred images was conducted on the basis of three different image quality metrics (IQMs): blind referenceless image spatial quality evaluator (BRISQUE) [93], natural image quality evaluator (NIQE) [94], perception-based image quality evaluator (PIQE) [95] and no-reference perceptual blur metric (NRPBM) [96]. Chosen no-reference image quality scores generally return a non-negative scalar. The BRISQUE score is in the range from 0 to 100. Lower score values better reflect perceptive qualities of images. The NIQE model is trained on a database of pristine images and can measure the quality of images with arbitrary distortion. The NIQE is opinion unaware and does not use subjective quality scores. The trade-off is that the NIQE score of an image might not correlate as well as the BRISQUE score with human perceptions of quality. Lower score values better reflect the perceptive qualities of images with respect to the input model. The PIQE score is the no-reference image quality score, and it is inversely correlated to the perceptual quality of an image. A low score value indicates high perceptive quality, and high-score values indicates low perceptive quality. The NRPBM is ranging from 0 to 1, which are, respectively, the best and the worst quality in terms of blur perception.

Table 5. Calculated image quality evaluator scores.

Kernel	Blur				Deblur
	BRISQUE	NIQE	PIQE	NRPBM	BRISQUE	NIQE	PIQE	NRPBM
0	32.5371	3.3942	20.1403	0.2464	32.5371	3.3942	20.1403	0.2464
5	40.1396	5.3276	35.7262	0.4279	39.0127	4.8747	24.6537	0.3461
10	41.0298	5.9315	62.2739	0.5734	42.9720	4.9778	24.1605	0.3548
15	41.9970	5.8995	71.8718	0.5927	43.1827	5.1326	28.7600	0.3461
20	42.3787	5.9203	76.7909	0.5951	43.1036	5.1680	29.8516	0.3527
25	42.6260	5.9928	79.3120	0.5933	43.7847	5.1275	30.7244	0.3613
30	42.8275	5.8444	80.7251	0.5880	45.2656	4.9797	31.9076	0.3775
35	42.8824	5.9419	81.4176	0.5820	45.8312	4.8937	34.4529	0.3956
40	43.0532	5.9418	82.2514	0.5741	46.0060	4.9044	35.9615	0.3899
45	43.1581	5.9602	82.6866	0.5662	46.4773	4.9784	35.7911	0.3712
50	43.1872	5.9524	83.2577	0.5573	46.7075	5.0238	36.4373	0.3521

shows calculated results of the aforementioned image quality evaluators. Figure 6 presents quality evaluation in graphical form.

The analysis of the results shown in Figure 6 emphasizes a clear improvement in image quality after the application of this method. The PIQE and NRPBM indices recorded a particularly significant improvement. The PIQE index correlates with the image perception by humans. Therefore, the visual, subjective image quality improvement presented in Figure 5 was confirmed by the PIQE index. The NRPBM index, especially intended for an objective blur evaluation in images, clearly points to a significant improvement in the quality for all data sets, with the greatest improvement for K = 15 and K = 20.

Photos subjected to the DeblurGANv2 method were used as a base to develop successive, typical photogrammetric products. This process followed the same procedure and processing settings, as the ones applied for generating reference products and photos with blur. The result was 10 products generated using synthetic photos, generated by neural networks.

Table 6. Reported survey data for a deblurred dataset.

Deblur	Flying Altitude (Reported) (m)	Ground Resolution (cm/pix)	Tie Points	Projections	Reprojection Error (pix)
0	74.5	2.45	416,194	1,515,694	1.09
5	75.1	2.69	442,180	1,645,007	1.04
10	75.3	2.69	457,941	1,665,776	1.01
15	74.7	2.7	453,347	1,624,915	1.07
20	72.8	2.7	437,927	1,497,823	1.16
25	79.8	2.66	433,953	1,422,424	1.19
30	79.8	2.65	425,342	1,328,750	1.25
35	79.7	2.65	411,533	1,220,397	1.29
40	79.7	2.65	387,720	1,109,842	1.32
45	79.6	2.65	344,131	970,994	1.4
50	79.1	2.63	300,779	834,237	1.57

presents basic study data based on photos without blurring.

Table 7. Average camera location error (X—Easting, Y—Northing, Z—Altitude).

Deblur	X Error (m)	Y Error (m)	Z Error (m)	XY Error (m)	Total Error (m)
0	1.01606	2.70717	112.933	2.89156	112.97
5	1.04393	2.67472	113.69	2.87122	113.726
10	1.04606	2.67152	113.843	2.86902	113.879
15	1.02665	2.7201	112.912	2.9074	112.95
20	0.952102	2.79801	111.177	2.95557	111.216
25	1.21931	2.51578	118.919	2.79569	118.952
30	1.22138	2.49247	118.929	2.77563	118.961
35	1.21005	2.47763	118.963	2.75733	118.995
40	1.1795	2.56589	118.892	2.824	118.925
45	1.16576	2.58083	118.88	2.83191	118.914
50	1.12633	2.56772	118.647	2.80389	118.68

shows a mean camera location error, and

Table 8. Control points RMSE (X—Easting, Y—Northing, Z—Altitude).

Deblur	Count	X Error (cm)	Y Error (cm)	Z Error (cm)	XY Error (cm)	Total (cm)
0	6	1.35198	0.928474	0.14282	1.6401	1.6463
5	6	1.52361	1.04413	0.16942	1.84705	1.85481
10	6	1.67629	1.13808	0.200033	2.02612	2.03597
15	6	1.53698	1.06043	0.183753	1.8673	1.87632
20	6	1.12107	0.638714	0.159253	1.29025	1.30004
25	6	1.13649	0.865287	0.184179	1.4284	1.44023
30	6	0.959562	0.893345	0.152415	1.31104	1.31987
35	6	0.75247	0.752481	0.133251	1.06416	1.07247
40	5	6.13895	6.27537	1.35764	8.77878	8.88314
45	5	4.35626	4.75946	1.36225	6.45209	6.59433
50	5	3.36417	3.03876	1.38842	4.5334	4.74125

the RMSE calculated for control point location. In this case, the points were also marked until the moment, when it was impossible to recognize them in the photos and possible to mark their center or the probable GCP center. It should be stressed that the operator was able to indicate all six control points for each blur kernel. They were possible to find and identify on all data sets.

Figure 7 graphically presents a comparison of all reported survey data for blurred and deblurred data set. The subfigures are presented in the following order: the number of tie points, the number of projections, reprojection errors, camera location total error, GCP total RMSE and the number of marked GCPs.

3. Results

This chapter presents and analyzes photogrammetric products developed both using blurred photos, as well as their corresponding deblurred photos. Figure 8 shows a fragment of a developed textured surface model. It should be noted that the texture was generated in each case, and it covered the surface model (mesh) in the right place. The texture quality is responsible for the quality of source photos. The blur visible in the photos is also visible on the surface model. As with the photos themselves, a clear mapping of the borders between sharp edges can be observed. The texture of anthropogenic models is clear, and a significant visual improvement was recorded in this regard. Such conclusions can also be directly derived from the parametric evaluation of the photos in Figure 6.

A textured digital surface model does not enable visually assessing the model geometric quality. For this purpose, Figure 9 shows a digital surface model in a uniform color. First of all, it should be noted that the terrain surface was mapped with geometry in models developed using photos without blur. Cars visible in this fragment have clearly marked edges. In the case of a blurred model, one can notice a lack of model generating correlation and stability, along with a blur kernel increase. For K = 15 and K = 50, the models of smaller objects on the surface are not mapped in practice.

As stated earlier, a significantly improved human perception of the photos is achieved. This allows us to conclude that the interpretive quality of an orthoimage generated using such photos will correlate with the quality of source photographs. Figure 10 shows fragments of developed orthophotomaps. A clear improvement in the interpretive quality of such a product can be observed in such images. Significantly blurred objects, e.g., due to their motion, are much clearer on an orthophotomap. This proves that enhancing source photos with the DeblurGANv2 methods translated directly to improving the interpretive quality of an orthophotomap.

The geometric quality of developed relief models was evaluated using the methods described in [30,97]. An M3C2 distance map (Multiscale Model to Model Cloud Comparison) was developed for each point cloud. The M3C2 distance map computation process utilized 3-D point precision estimates stored in scalar fields. Appropriate scalar fields were selected for both point clouds (referenced and tested) to describe measurement precision in X, Y and Z (sigmaX, sigmaY and sigmaZ). The results for sample blurs are shown in Figure 11. As can be seen in the presented images, in the case of models based on photos without blur, the distance of a tested model to a reference cloud is stable and there are no significant differences, which is depicted by an even green color over the entire area. In the case of images with blur, as might have been expected, the geometry of the entire surface has been significantly modified. Model deformation, resulting from the systemic errors occurring during aerotriangulation (so-called doming error), increases together with an increasing blur kernel and adopts values of up to 50 m, at the boundaries of the modelled surface.

The statistical evaluation of calculated M3C2 distances for all variants was shown in

Table 9. Calculated M3C2 distances.

Kernel	Blur		Deblur
	Mean	Std. Deviation	Mean	Std. Deviation
5	−0.047	0.7171	−0.0415	0.6335
10	−0.0732	0.8571	−0.0495	0.6745
15	−3.2327	13.6939	−0.0074	0.8452
20	1.0668	12.6647	0.0464	0.8117
25	−0.288	1.0104	−0.2643	0.8803
30	0.1825	1.0667	−0.285	0.9579
35	0.3792	1.1389	−0.3187	1.0308
40	−0.1491	1.0717	−0.2452	1.0806
45	0.1205	2.2933	−0.2366	1.1688
50	−3.7489	14.1588	−0.2758	1.1695

and its graphic form in Figure 12. The statistical distribution of M3C2 distances is close to normal, which means that a significant part of the observations is concentrated around the mean. The mean and standard deviations for blurred images are not stable and change drastically depending on the blur kernel. The drastic change is observed for 15, 20 and 50 kernels. High stability can be observed for models generated from deblurred photos. The mean varies around zero and the standard deviation grows steadily, to take twice the initial value for the highest blur kernel.

4. Conclusions

The results presented in the previous section show that the method for improving blurred photo quality significantly contributed to an improvement in the general quality of typical photogrammetric products. The model quality can be analyzed in terms of their geometric accuracy and interpretive quality.

The geometric accuracy of the models generated from photos without blur was maintained, which is evidenced by the low means and standard deviation of the compared models. This deviation steadily increases with a growing blur kernel. There was also no significant doming error in these models, which leads to the correct generation of a digital surface model, which preserves severe altitude changes in object edges. As has been shown, object edges in blurred photos fade away at higher blur kernel values. The proposed method enables emphasizing these edges, which consequently retain their geometry.

The interpretive quality of textured products and photos clearly increased. It has been shown, beyond any doubt, that blur reduction significantly improves image perception, and the objects depicted in an orthoimage are clearer. According to the author, this property has great potential for practical application, especially for the reconstruction of highly urbanized places and objects. Blur reduction was higher in image areas with strongly marked non-natural objects (buildings, roads, cars, etc.). This fact largely results from the nature of the photos used for neural network model training. The majority were photos showing fragments of urbanized areas. This property strongly improves the interpretive properties and can be used to develop the maps of urban areas, as well as to extract land cover elements from an orthophotomap.

In the case of the used GCPs (plate with a 30 cm × 30 cm graphic symbol), it was possible to indicate them, even in photos with an initially highest blur. It certainly improved the aerotriangulation outcome and the geometry of the developed models. However, the visibility of these points was not fully maintained, which can constitute grounds to further research of this method. At this point, one can lay a hypothesis that by training networks using photos of similar GCPs or an image of typical GCPs used in photogrammetry, it is possible to enhance and obtain a significant improvement in blur reduction in images showing ground, signaled control points.

The method for improving the quality of blurred photos applied in this research paper has been tested using typical photogrammetric software. Surprisingly, the software turned out to be rather blur resistant and enabled generating models based on all test data, even the ones with the highest blur. It is a fact that the surface of the modelled terrain was deformed; nevertheless, the course of the process of developing a model based on blurred photos was typical. Both the geometric, as well as the interpretive quality of developed photogrammetric models were improved using the neural method for reducing blur, based on generative adversarial networks.

In the conventional photogrammetric process, blurred images are completely unusable. In specific cases, it may turn out that all images have visible blur for the entire photogrammetric flight. This situation means that it is very difficult to locate GCPs, object details and edges are unclear and fading, and images have a weak interpretation quality. The application of the method will eliminate any blurred images and, consequently, it will be possible to use the acquired data for the modelling. In photogrammetric software, which is commonly employed and particularly in their cloud versions, it will be possible to implement an additional feature, which will eliminate unwanted blur. The performance of the method based on neural networks is relatively fast, which means that its application will not significantly slow down the process of building photogrammetric models. Additionally, the application of the method significantly expands the capabilities of the photogrammetric software, due to its universality and independence from the blur character.