Super-Resolution Techniques in Photogrammetric 3D Reconstruction from Close-Range UAV Imagery

Abstract

Current Multi-View Stereo (MVS) algorithms are tools for high-quality 3D model reconstruction, strongly depending on image spatial resolution. In this context, the combination of image Super-Resolution (SR) with image-based 3D reconstruction is turning into an interesting research topic in photogrammetry, around which however only a few works have been reported so far in the literature. Here, a thorough study is carried out on various state-of-the-art image SR techniques to evaluate the suitability of such an approach in terms of its inclusion in the 3D reconstruction process. Deep-learning techniques are tested here on a UAV image dataset, while the MVS task is then performed via the Agisoft Metashape photogrammetric tool. The data under experimentation are oblique cultural heritage imagery. According to results, point clouds from low-resolution images present quality inferior to those from upsampled high-resolution ones. The SR techniques HAT and DRLN outperform bicubic interpolation, yielding high precision/recall scores for the differences of reconstructed 3D point clouds from the reference surface. The current study indicates spatial image resolution increased by SR techniques may indeed be advantageous for state-of-the art photogrammetric 3D reconstruction.

1. Introduction

Image data can be exploited by SfM (Structure-from-Motion) and MVS (Multi-View Stereo) photogrammetric tools towards reconstructing robust, free of outliers, and elaborate 3D models [1,2]. Essentially, these image-based approaches start from aerial and/or terrestrial image blocks to automatically perform image registration (via sparse image matching) and subsequently reconstruct accurate textured 3D models of the depicted scenes (via dense image matching). Since these automated pipelines are capable of generating high-quality spatial data at a reasonable cost, the now widely available software tools for SfM/MVS photogrammetry have triggered a tremendous increase in the number of its applications, notably in the geosciences but also in all possible fields requiring reliable 3D geospatial information. Consequently, the SfM/MVS photogrammetric tools also attract today unprecedented numbers of mostly non-expert users, namely those with little or without formal training in photogrammetry [3].

Clearly, however, this striking popularity of photogrammetric 3D reconstruction approaches is largely due to an ever-growing variety of autonomous platforms for flexible image acquisition known as unmanned aerial vehicles (UAVs) or drones. Thanks to their obvious advantages, both financial and technical, against conventional satellite and manned aircraft platforms [4], UAVs have become the main camera platform currently used for the purposes of aerial mapping. A summary literature review of recent developments and applications involving light-weight UAVs and SfM/MVS software is given in [5].

This is also true for the specific environment of cultural heritage, where UAVs have provided an attractive alternative to traditional, but unwieldy, camera platforms for low-altitude image acquisition (balloons, kites, elevators, tripods). In a review of their use in this field [6], it is noted that, beyond mere technical innovation, the methodological novelty of UAVs lies mainly in allowing archaeologists to exercise direct control over all aspects of survey processes (platform, sensors, and processing of collected data). As a consequence of the above, the UAV/SfM/MVS photogrammetric approach is now also very popular in the field of cultural heritage. The recent detailed review [7] discusses state-of-the-art tools and methodologies for image acquisition, data processing, and 3D reconstruction in relation to cultural heritage.

On the other hand, in multiple applications of Remote Sensing (RS), such as surveillance and satellite imaging practices, the spatial resolution of aerial or satellite images has a strong impact on the precision and reliability of the information which is extracted from the images [8,9,10]. High-resolution (HR) images may of course be acquired under controlled scenarios in the absence of related constraints on the recording hardware. However, in certain instances the MVS input images are of limited spatial resolution (as in the case of historic photographic archives); the hardware may impede the collection of the required data amount; power consumption may in turn constrain the hardware, e.g., in the case of drones [11]. As regards imagery from unmanned aerial vehicles (UAVs) in particular, issues of low spatial resolution are mainly attributed to the flight height and the selected resolution in 3D space (ground element or “groundel”), especially in urban environments or industrial and construction sites, as well as to their generally low-cost, consumer-grade digital RGB cameras. The problem is that requirements of safe flight and/or the camera parameters do not always conform with the needed ground sampling distance, geometry or texture quality; in such cases, the generated 3D model will most likely be poor in detail and completeness, regardless of the adopted MVS algorithm [11,12,13]. For straightforwardly super-resolving a Digital Surface Model (DSM), on the other hand, a large amount of data is required for model training [14]. Hence, an algorithmic increase of image spatial resolution may indeed be of high interest as a means for effectively improving not only the visual quality but also the metric accuracy of results. Super-Resolution (SR) algorithms upsample original low-resolution (LR) imagery in an attempt to improve both their visual and metric properties, enrich detail and possibly recover high image frequencies [10,15,16,17,18,19,20,21,22,23,24,25]. In this sense, single-image super-resolving techniques, in particular, seem to be suitable for improving 3D reconstruction, thus calling for a closer examination in terms of effectiveness.

Ιn this work, a study is carried out on different image SR techniques to assess their suitability regarding their inclusion in a 3D reconstruction procedure. SR techniques are validated on UAV-captured image data. In particular, the SR algorithms presented in [19,22,23,24] are examined as means for super-resolving images which enter the 3D reconstruction task in the field of cultural heritage. The MVS task is performed by using the Agisoft Metashape photogrammetric tool [26]. The specific contribution of our work is a thorough investigation of the suitability of different pre-trained image SR techniques applied on oblique low-altitude (close-range) imagery for the purposes of photogrammetric full 3D reconstruction (as generally required in the field of cultural heritage projects).

This work is organized into 6 sections. Section 2 presents the literature on improving 3D reconstruction by means of SR tools. Section 3 describes the image datasets, the SR techniques, the 3D reconstruction approach and the methodology adopted for the comparisons. The results are given in Section 4, while Section 5 discusses the main issues arising throughout the current study. The conclusions are drawn in Section 6.

2. Literature Review of 3D Reconstruction with the Aid of Super-Resolution

SR techniques can algorithmically increase the spatial resolution of images (but it is noted that, while thereby details get clearer and more discernible, artefacts may also be induced). The exploitation of SR tools for enhancing 3D reconstruction procedures is an open research field, since relatively few related studies have appeared in the literature up to now [11,12,13,14,25,27,28].

Working in the context of conventional mapping and DSM generation from satellite imagery, the authors in [14] examined the introduction of different refined SR models aiming at a workflow for creating DSMs per subpixel level. Qualitative analysis and a summary statistical measure (RMS difference in DSM elevations) led to the conclusion that a subpixel level DSM of higher reliability may be more effectively obtained via image SR techniques rather than bicubic image interpolation or direct DSM upscaling.

In Ref. [25] the neural networks RDN [29] and ESRGAN [30], along with bicubic interpolation, were evaluated regarding DSM generation in urban areas from pairs of satellite images. Judging mainly from the summary quantitative measures of DSM differences, these authors found no evidence that the employment of image SR networks outperforms standard bicubic image interpolation for the investigated 3D task. In a more detailed local investigation on stereo matching, image disparities at the maxima of the distributions of the employed similarity measure (cross-correlation coefficient) along corresponding epipolar lines of satellite stereo pairs were evaluated against ground truth elevation data from Lidar. It was seen that matching benefited strongly from SR for features of high contrast; on the other hand, in uniformly textured areas SR turned out to be disadvantageous, accompanied by uncontrolled artefact occurrence and inconsistent patterns which led to poor matching in such areas (not rare in urban environments). Concluding, the authors skeptically wondered whether, without addressing such shortcomings, one may actually exploit SR potential with reliability in 3D photogrammetry. One may remark, however, that (unlike standard stereo matching) multi-view matching may provide more means to efficiently exclude outlying image content and SR-induced image artefacts.

The use of image SR algorithms has been suggested also in [12] for cases where a required higher ground resolution cannot be possibly met by shorter imaging distances or different cameras/lenses. In this work, photogrammetric products (from the Agisoft Metashape tool) from vertical images captured by a small commercial UAV were evaluated against the same data pre-processed with a method based on the SR generative adversarial network (SRGAN) [31]; the data collected at a lower altitude served for ground truth. It was concluded that photogrammetric products created after SR-processing of images from larger imaging distances presented quality close to those from smaller distances (and, additionally, avoided certain occlusion issues met in imagery from smaller heights but with inadequate multiple overlaps).

A recent paper [11] also investigated how an increase in resolution by SR techniques of the images which enter an MVSNet algorithm is reflected in quality improvements of the reconstructed 3D models. COLMAP [32] and CasMVSNet [33,34] were the 3D reconstruction algorithms used. In order to increase the image spatial resolution by a factor of 2, the SR algorithm Deep Back-Projection Networks (DBPN) [35], as well as bicubic interpolation, were tested. It was shown that the introduction of a SR stage before recovering the depth maps led in most cases to a better 3D model in the case of both PatchMatch-based and Deep-Learning (DL) based algorithms. Overall, it was concluded that SR especially improves the completeness of reconstructed models, turning out to be particularly effective in the case of well-textured scenes.

The potential of a SR model for single images, based on the ESRGAN deep convolutional neural network [26], to improve the spatial resolution of UAV-captured RGB images was investigated in [13]. The model was retrained using HR images and corresponding LR images created by downsampling the original HR images. A qualitative and quantitative assessment of results from a photogrammetric Structure from Motion (SfM) process (again with the Agisoft Metashape software) indicated that the internal and external camera parameters of the SR images presented values close to those from the adjustment of the HR images. It was also concluded that SR resampling is beneficial for scene reconstruction; a more thorough evaluation, however, would have allowed a more detailed comparison of the two dense 3D models.

Finally, other recent applications employed a SR technique to assist road pavement monitoring with UAV images [27] and to improve building façade 3D reconstruction using images from a smartphone [28]. In the different context of consumer RGB-D cameras, the approach in [36] aims at high-quality 3D reconstruction involving, among other steps, the application of a depth super-resolution network directly on noisy, low-resolution depth images combined with a high-resolution intensity image of the scene (for a review of similar methods for super-resolving depth images from RGB-D cameras see [37]).

3. Materials and Methods

3.1. Low-Altitude Image Data

The dataset in our low-altitude close-range photogrammetric study consists of 29 UAV-captured oblique images and depicts Omorfokklisia, a 13th-century Byzantine church on the outskirts of Athens, Greece. The UAV platform was a DJI Phantom 3 Professional with a 12 MP camera (pixel size: 1.6 μm; nominal focal length: 3.61 mm). The mean flying height was ~18 m and the camera tilt angle ~45°. A circular flight path in the “point of interest flight mode” was chosen, i.e., the UAV orbited automatically around the object of interest. Figure 1 presents two typical images of the data set, which also show the environment of the monument.

3.2. Deep-Learning Based Super-Resolution Techniques

Besides conventional bicubic image upscaling per factor 4 (used here for the purposes of comparison), the following DL-based SR techniques have been tested in the context of 3D reconstruction. All algorithms have been run on Kaggle [38].

• RankSRGAN: This SR technique [23] was employed for image resolution enhancement by a factor of 4. The RankSRGAN framework is structured into 3 stages. In stage 1, various SR techniques serve for super-resolving images of public SR datasets. Next, pair-wise images are ranked based on the quality score of a chosen perceptual metric while the corresponding ranking labels are preserved. Stage 2 regards Ranker training. The learned Ranker has a Siamese architecture and should be able to rank images depending on their perceptual scores. In stage 3 the trained Ranker serves as the definition of a rank-content loss for a typical SRGAN to generate visually “pleasing” images.
• Densely Residual Laplacian Super-Resolution (DRLN): This SR technique [19] also served in this study for SR by a factor of 4. It relies on a modular convolution neural network, where several components that boost the performance are employed. The cascading residual on the residual architecture used facilitates the circulation of low-frequency information so that high and mid-level information can be learned by the network. There are densely linked residual blocks that reprocess the previously computed features, which results in “deep supervision” and learning from high-level complex features. Another significant characteristic of the DRLN technique is Laplacian attention. Via the latter, the crucial features are modeled on multiple scales, whereas the inter- and intra-level dependencies between the feature maps get comprehended.
• Hybrid Attention Transformer Super-Resolution (HAT): The images were again super-resolved by a factor of 4 via this SR technique [24]. HAT is inspired by the fact that transformer networks can greatly benefit from the self-attention mechanism and exploit long-range information. Shallow feature extraction and deep feature extraction precede the SR reconstruction stage. The HAT transformer jointly utilizes channel attention and self-attention schemes as well as an overlapping cross-attention module. This SR technique aims at activating many more pixels for the reconstruction of HR images.

Finally, in our experiment, the Holistic Attention Network (HAN) [22], in which the feature representation for SR is advanced, has also been used. There are two attention modules jointly working to successfully model informative features among hierarchical layers. The layer attention module learns the weights for hierarchical features by considering correlations of multi-scale layers, channels, and positions, while a channel-spatial attention module comprehends the channel and spatial interdependencies of features in each layer. However, the HAN technique needs input images with an equal number of rows and columns, which is not the case here. It has been tested in other experiments with satisfactory performance, but no results can be shown in this work.

3.3. SfM/MVS Tool

Several alternative software tools are currently available to end-users for the photogrammetric generation of complex 3D models from unordered imagery. Although most of them are based on implementing the standard SfM procedure, they demonstrate variations in terms of user-friendliness and interactivity, acquisition cost (open-source or commercial software), level of customization, and overall processing time. Some of the most commonly used open-source SfM software options include COLMAP [39] and Meshroom [40]. In the commercial market, on the other hand, the tools ContextCapture [41], Agisoft Metashape [26], RealityCapture [42], and Pix4DMapper [43] are some of the dominant players. For the 3D reconstruction stage in the present contribution, the photogrammetric 3D reconstruction pipeline of Agisoft Metashape was employed due to its wide popularity (particularly among non-experts in photogrammetry) in both the remote sensing and the cultural heritage communities as well as its ease of use. Note that this software is also available for use under an academic license.

It may be mentioned that the Cascade Cost Volume Multi View Stereo CasMVSNet [33,34], a DL based algorithm for MVS reconstruction, has also been experimented with; regrettably, it ran out of memory when asked to process the HR images.

3.4. Methodology for 3D Model Reconstruction

Starting from digital images, Agisoft Metashape produces 3D spatial data in an accurate and fast manner. Characteristic feature functions of this image-based reconstruction software include point extraction and matching, photogrammetric bundle adjustment, dense point cloud editing and classification, DSM generation, georeferenced orthomosaic generation, stereoscopic measurements, 3D model generation and texturing, along with panorama stitching.

Similar to other publications [13,25], bicubic downscaling per a factor of 4 has been applied to the original HR images to generate their LR counterparts. Then, the three mentioned SR tools RankSRGAN, DRLN, and HAT [19,23,24] were used to bring these LR images back to the original spatial resolution, thus producing the super-resolved (SR) images; for comparison purposes, bicubic interpolation was also applied for upsampling the LR images by a factor of 4. In Figure 2 typical image patches are seen.

The super-resolved image has a clearly improved quality (edges have been enhanced, yet poorly textured areas have been somewhat “flattened” compared to the HR images). No discernible SR-induced artefacts are to be observed.

The original HR images were then input into the Agisoft tool for full self-calibrating bundle adjustment (estimation of interior and exterior camera orientations) and 3D reconstruction. The resulting HR 3D model served as the reference model in comparisons. Next, SR 3D models were generated from the SR images, along with a 3D model from the bicubically upscaled images; finally, the LR images produced the low-resolution 3D model. All high-precision reconstruction processes in Agisoft were performed with the same parameters. In summary, the following six models were generated:

(a)

one model from the original 3000 × 4000 high-resolution images (HR)

(b)

one model from the downsampled 750 × 1000 low-resolution images (LR)

(c)

one model from the bicubically upsampled 3000 × 4000 low-resolution images (BU)

(d)

three models from the super-resolved 3000 × 4000 low-resolution images (SR).

3.5. Criteria for the Comparison of Point Clouds

The evaluation of the similarity between point clouds has generally two distinct steps: the registration of the clouds into the same reference (geodetic) system; and the comparison itself. The objective of registration is basically to calculate the 6-parameter rigid (translation and rotation) or the 7-parameter similarity (plus uniform scaling) 3D transformation for point clouds of arbitrary initial position in order to express them in a common coordinate system. In general, the registration step includes, in turn, two distinct steps: coarse registration (initialisation); and registration refinement.

The alternatives for point cloud registration have been extensively reviewed in the recent literature [44,45,46,47,48,49], particularly in the context of photogrammetric and LiDAR 3D reconstruction. A large part of the reviewed research is dedicated to the automatic establishment of distinctive feature point correspondences for initializing registration, among which DL-based approaches are being currently investigated. In our case, however, initialisation presents no problem since ground control points (GCPs) have been used, which additionally gives to all 3D models a common scale. Alternatively, one could use the GPS information for the location of camera stations; or else (since in our case the corresponding images of the different image sets share the same exterior camera orientation) one could simply transform these orientations into a common reference system.

On the other hand, a substantial part of research on point cloud registration addresses the general case of “non-cooperative” datasets of point clouds stemming from different sources, where the registration task may face varying levels of noise and types of outliers, considerable scaling differences, very different point densities, and only partially overlapping 3D data. In cases (like here), however, of point clouds from the same source, with equal point density (excepting the low-resolution 3D model), no scaling issues, total overlap, similar levels of noise, and good initial registration, methods based on the ICP (Iterative Closest Point) registration algorithm [50,51] are generally expected to perform well [46,49,50]. Furthermore, considering 3D rigid transformations (rotation plus translation), the transformation which minimizes the total Euclidean distance between all point correspondences is usually regarded as being optimal [48].

Coming, finally, to the comparison itself between registered point clouds and the evaluation of their quality, several different approaches have been reviewed in the context of aerial photogrammetry and laser scanning [52,53]. Both these reviews conclude that, among other possible metrics, in such applications, the most popular and representative evaluation option for cloud-to-cloud comparisons and noise evaluation appears to be the so-called point-to-plane approach (see also [54]). This means that the evaluation measure for the difference between point clouds is the distance of a reconstructed point from the reference surface, which is modelled locally by fitting a mathematical primitive (in this case a plane) on the nearest point and its immediate neighbourhood.

3.6. Comparison of Point Clouds

Pairwise comparisons of the dense 3D point clouds were thus performed, namely the LR model, the BU model, and the three SR models were each compared against the reference HR model. Based on the above considerations, we used the popular 3D point cloud processing software CloudCompare [55], which allows for the calculation of distances between “corresponding” cloud points by means of the ICP algorithm. For each point of a cloud, its corresponding point was determined here as its projection on the plane fitted to its 6 closest points of the other cloud.

Following [56,57], reconstructions were evaluated in terms of precision p and recall (completeness) r over four distance thresholds t: 0.5, 1, 2, and 3 cm. Completeness is generally defined as the percentage of ground truth points (here: those of the reference model) for which the distance to their corresponding (as defined above) point in the reconstructed cloud is below the selected evaluation threshold. Precision (sometimes also referred to as accuracy), on the other hand, is defined as the percentage of reconstructed points whose distance from their corresponding (as defined above) ground truth points falls within the selected threshold.

Thus, precision quantifies the metric quality of the reconstruction: how close are the reconstructed points to ground truth; recall, in turn, quantifies the completeness of reconstruction: to which extent the ground-truth points are covered [56]. Obviously, the properties of precision and recall (completeness) are both crucial for assessing quality of reconstructions. Hence, they are often combined in the F-score, which represents a summary quantitative measure, being the harmonic mean of the precision p and recall r values: F = 2 × (p × r)/(p + r). The following Tables concentrate all values of p(t), r(t) and F(t) for the respective evaluation threshold t.

4. Results

4.1. Full 3D Model

The reconstructed 3D models were first cut to keep only the object of interest (in Figure 3 views of two such models are given). The numbers of reconstructed model points are seen in

Table 1. Number of point cloud vertices.

3D Models	Number of Vertices
HR (reference)	907,917
LR	87,844
BU	1,020,456
SR RankSRGAN	1,091,278
SR DRLN	1,012,209
SR HAT	951,992

.

Models from SR images have more vertices than the HR (reference) model; it remains to be seen whether they fit precisely to the latter. The 3D model from the LR images, on the other hand, has of course far fewer points (less than 10%).

Then, each of the reconstructed LR, BU, and SR point clouds were registered onto the properly scaled HR point cloud to allow for comparisons via point-to-plane distances (as previously defined) from it. No threshold for maximum distance values is used in these calculations.

Visualizations of model differences from the HR model are seen in Figure 4. Truncated histograms of differences between the BU and SR models (the differences in the LR model are very large, as already depicted in Figure 4) from the HR model are shown in Figure 5.

Detailed results for the registration of reconstructed-to-reference point clouds (precision p) and of reference-to-reconstructed point clouds (recall r), along with the respective F-score values (F), are seen in

Table 2. Precision (p), Recall (r) and F-score (F) percentage of the calculated distance differences for the full point cloud. Percentage values account for the points with distances which are smaller than, or equal to, the respective evaluation threshold t.

Point Cloud	Evaluation Threshold t (cm)
	0.5			1.0			2.0			3.0
	p	r	F	p	r	F	p	r	F	p	r	F
LR from Agisoft	15.52	13.79	14.61	30.40	27.13	28.67	57.06	51.49	54.13	77.76	72.09	74.81
BU	49.62	46.62	48.08	78.74	75.28	76.97	96.17	94.89	95.52	98.99	98.56	98.77
RankSRGAN	41.84	40.08	40.94	69.71	67.26	68.46	91.71	90.24	90.97	97.47	96.87	97.17
DRLN	54.38	53.04	53.70	82.80	81.35	82.07	97.12	96.85	96.98	99.19	99.18	99.18
HAT	56.16	55.10	55.62	84.25	82.94	83.59	97.35	97.17	97.26	99.22	99.27	99.24

. The number of points with distances from the reference model smaller than, or equal to, four selected evaluation thresholds t are presented. Results will be evaluated in the next section.

4.2. Model Segments

Next to the comparison of the 3D models of the entire monument, 3D reconstruction was also investigated in different individual model segments. In the mutual registration of the full (all-around) models, possible misalignments originating in the SfM step (errors of image interior/exterior orientation in bundle adjustment) may be reflected in stronger local surface deviations. The main purpose of this paper is to evaluate the effectiveness of different SR techniques by studying their impact on the multi-view reconstruction procedure rather than on the SfM step. Although of course the steps of SfM (sparse matching) and 3D reconstruction (dense matching) cannot be totally decoupled from each other, an independent registration of segments of the reconstructed 3D models to their reference counterparts might possibly mitigate the impact of such misalignments. The issue of SfM-induced 3D reconstruction problems will be discussed in the next section.

Thus, the same comparisons were made for the three monument segments A, B, and C seen in Figure 6. These represent three viewing directions, namely a top view (a complex of domes and slanted tile roofs) and two different side views (which mainly include planar surfaces but also tile roofs). As in the preceding section,

Table 3. Precision (p), Recall (r) and F-score (F) percentage of distance values for segment A.

Point Cloud	Evaluation Threshold t (cm)
	0.5 cm			1 cm			2 cm			3 cm
	p	r	F	p	r	F	p	r	F	p	r	F
BU	44.88	41.92	43.35	75.25	71.03	73.08	95.72	94.03	94.87	98.95	98.40	98.67
RankSRGAN	39.72	36.82	38.21	67.74	63.79	65.71	91.04	88.92	89.97	97.33	96.62	96.97
DRLN	54.43	52.56	53.48	83.16	80.98	82.05	97.23	96.77	97.00	99.23	99.16	99.19
HAT	55.89	54.68	55.28	84.56	82.71	83.62	97.59	97.18	97.38	99.28	99.28	99.28

,

Table 4. Precision (p), Recall (r) and F-score (F) percentage of distance values for segment B.

Point Cloud	Evaluation Threshold t (cm)
	0.5 cm			1 cm			2 cm			3 cm
	p	r	F	p	r	F	p	r	F	p	r	F
BU	50.83	47.37	49.04	80.38	75.93	78.09	96.32	95.09	95.70	99.05	98.78	98.91
RankSRGAN	40.11	38.10	39.08	68.15	65.36	66.73	90.91	89.29	90.09	97.17	96.74	96.95
DRLN	52.85	44.88	48.54	82.14	75.76	78.82	97.04	96.08	96.56	99.21	99.11	99.16
HAT	54.09	52.63	53.35	83.09	81.50	82.29	97.14	96.96	97.05	99.22	99.28	99.25

and

Table 5. Precision (p), Recall (r) and F-score (F) percentage of distance values for segment C.

Point Cloud	Evaluation Threshold t (cm)
	0.5 cm			1 cm			2 cm			3 cm
	p	r	F	p	r	F	p	r	F	p	r	F
BU	43.19	47.62	45.30	74.27	77.70	75.95	95.77	96.36	96.06	98.97	99.18	99.07
RankSRGAN	38.62	40.63	39.60	66.90	68.91	67.89	91.46	91.85	91.65	97.57	97.75	97.66
DRLN	42.27	53.96	47.40	72.30	82.94	77.25	95.12	97.51	96.30	98.89	99.45	99.17
HAT	48.40	54.94	51.46	78.68	84.15	81.32	96.62	97.78	97.20	99.13	99.44	99.28

concentrate the respective information from all segment model comparisons regarding the percentages of calculated distance values for the same evaluation thresholds.

5. Discussion

The results of the previous section show that, as expected, LR images are by far superseded by both BU and SR methods regarding both the number of points and precision/recall performance. Although bicubic interpolation offers substantial improvement versus LR images, it was expected that it would be outperformed by RankSRGAN; in this particular investigation, however, the bicubic interpolation appears to be clearly superior over this SR tool. The other two SR approaches (DRLN and HAT) gave the best results, with similarly high precision and recall percentages, thus appearing to be very efficient as regards both model accuracy and completeness. In fact, in all cases, less than 4% of all tested points differed by more than 2 cm from their expected position, and less than 1% by more than 3 cm. An impression given by Figure 4 is that DRLN gives the best outcome; yet the tabulated results revealed that HAT consistently gives (marginally) better p, r, and F values. In fact, both methods appear here to be equivalent in improving 3D reconstruction when starting with LR images. Furthermore, DRLN and HAT upscaling produced more reconstructed 3D points than the original HR images (namely by 11.5% and 4%, respectively), which however appear to conform with the used reference surface.

Of course, the final result of 3D reconstruction pipelines carries the impact of multiple intermediate steps; error propagation through them is a topic not fully mastered in photogrammetry, as rightly remarked in [25]. In typical cases, for instance, sparse matching (SfM) and dense matching (MVS) errors are intertwined in an almost inseparable way; hence, it is difficult in our case to separately assess the impact of SR tools on each individual step. The study of local similarity measures along epipolar lines, as in [25], may indeed give a picture for the performance of SR tools in stereo matching (although, in the general case, the positions of epipolar lines themselves have already been affected by bundle adjustment). But when more overlapping views (and thus more homologous epipolar lines) are available, the situation is far more complicated, the more so since outlying local contributions of individual images can be in this case effectively discarded.

On the other hand, a straightforward comparison of the values of image orientation parameters with those from a different solution may be an indication of their actual closeness; this led in [13] to the conclusion that interior and exterior imaging geometry could be accurately retrieved from SR image sets. These parameters, however, may be strongly intercorrelated (especially in geometrically weak image networks) and, if necessary, call for more thorough investigations. In any case, strong imaging configurations are needed in projects where accuracy is a central issue.

Indeed, the robustness of a bundle adjustment heavily depends on image network geometry; for instance, image tilts and cross-flights (which allow significant image roll angles) are generally desirable. Unlike typical networks of vertical images mostly used in publications on SR-based 3D reconstruction, a strong geometry has been adopted here, based on oblique UAV imagery pointed towards the object of interest, which guarantees high numbers of image ray intersections in 3D space. It might be partly attributed to this fact that our results for the full 3D model are similar to those for the model segments.

6. Conclusions

Deep-learning-based image SR techniques have been proven as a powerful means for enhancing image spatial resolution, visual quality, and detail. Our work shows that it can also contribute to improving 3D scene reconstruction. Corroborating the results of other relevant publications (based on satellite or vertical UAV images), the present study, using low-altitude oblique UAV images and pre-trained SR tools, clearly indicated that state-of-the-art photogrammetric 3D reconstruction can give improved results by exploiting the increased image spatial resolution thus provided. Two of the three SR tools led to high and almost equivalent accuracy and completeness scores in 3D reconstruction when compared with models from images of original high resolution.

Future work includes experimentation with results from model retraining; evaluation of different imaging geometries for SR-based 3D reconstruction; also, research regarding the possibility of isolating the effects of SR techniques on the basic steps of a SfM/MVS pipeline (for sparse point extraction and matching).

As in most relevant publications, the low-resolution images here were not original images but bicubically downscaled versions of the originals, which might introduce a certain bias in the comparisons. An alternative would be the employment of different imaging distances, as in [12], to provide LR and HR data. But this also presents problems since such images are not directly comparable regarding 3D reconstruction due to their different image perspectives. Thus, we also intend to perform tests with images acquired with different resolutions but under similar geometric conditions, namely from equal imaging distances and with the same camera tilts.