A Dynamic Multi-Projection-Contour Approximating Framework for the 3D Reconstruction of Buildings by Super-Generalized Optical Stereo-Pairs

Abstract

In this paper, a novel framework of the 3D reconstruction of buildings is proposed, focusing on remote sensing super-generalized stereo-pairs (SGSPs). As we all know, 3D reconstruction cannot be well performed using nonstandard stereo pairs, since reliable stereo matching could not be achieved when the image-pairs are collected at a great difference of views, and we always failed to obtain dense 3D points for regions of buildings, and cannot do further 3D shape reconstruction. We defined SGSPs as two or more optical images collected in less constrained views but covering the same buildings. It is even more difficult to reconstruct the 3D shape of a building by SGSPs using traditional frameworks. As a result, a dynamic multi-projection-contour approximating (DMPCA) framework was introduced for SGSP-based 3D reconstruction. The key idea is that we do an optimization to find a group of parameters of a simulated 3D model and use a binary feature-image that minimizes the total differences between projection-contours of the building in the SGSPs and that in the simulated 3D model. Then, the simulated 3D model, defined by the group of parameters, could approximate the actual 3D shape of the building. Certain parameterized 3D basic-unit-models of typical buildings were designed, and a simulated projection system was established to obtain a simulated projection-contour in different views. Moreover, the artificial bee colony algorithm was employed to solve the optimization. With SGSPs collected by the satellite and our unmanned aerial vehicle, the DMPCA framework was verified by a group of experiments, which demonstrated the reliability and advantages of this work.

1. Introduction

In recent years, there has been a dramatic proliferation of research concerned with three dimensional (3D) reconstruction of buildings by remote sensing resources. Data collected by different sensors could be used finishing this work, like optical images [1], synthetic aperture radar (SAR) [2], and light detection and ranging (LiDAR) [3]. There are also some multi-data source and multi-sensor approaches [4,5]. Although higher precision 3D points of a city area can be obtained by LiDAR, it is an inadequate resource for certain applications, such as rescue and anti-terrorism reconnaissance. Undisputedly, optical images are the most adequate remote sensing resources. Super-generalized stereo-pairs (SGSPs) can be defined as two or more optical images, which have overlapping coverage areas and might have different resolutions, collected by the same or different sensors at diverse times and views. As we all know, a classic 3D reconstruction framework is based on stereo-pairs collected by spaceborne or airborne [6,7]. These stereo-pairs are required to satisfy certain restrictions (convergence angle, base-to-height ratio, etc.) to perform better stereo matching and calculation of 3D coordinates [8]. However, SGSPs might not afford these restrictions, and stereo matching is undoubtedly the technical bottleneck when doing 3D reconstruction by SGSPs [9,10]. People always failed to generate digital elevation/surface model of regions of buildings, and cannot do further 3D shape modeling for the buildings. Although some gradually developing products, like smart 3D and Street Factory, generate dense 3D point clouds using state-of-the-art pixel-based multi-stereo image matching [11], an amount of manual correction is needed to enhance the geometric detail of 3D building models. In other words, the 3D reconstruction of buildings, using SGSPs, cannot be well achieved by traditional frameworks.

Here, we introduce a dynamic multi-projection-contour approximating (DMPCA) framework to reconstruct the 3D shape of buildings using SGSPs.

2. The Dynamic Multi-Projection-Contour Approximating Framework

The main framework of DMPCA is shown in Figure 1. The key idea is that in each of the specific views of SGSPs, we approximate the projection-contour of the building in the optical images by the projection-contour of a simulated 3D model of that building. Specifically, we establish an optimization to find a group of parameters of a simulated 3D model. It minimizes the total differences between projection-contour of a building in SGSPs and projection-contours of the simulated 3D model. Then, the simulated 3D model, defined by the optimized parameters, dynamically approximate the actual 3D shape of the building.

The work here is based on the hypothesis as expressed in Equation (1). Assume that a simulated 3D shape S_sim is close to the actual 3D shape of building S_true, when two or more projection-contours (PC) PC_SGSPs from actual images are approximated by the corresponding PC_sim (N is the number of images of SGSPs; PC_SGSP is the projection-contour of the building after imaging; PC_sim is the projection-contour of the building generated by S_sim).

$$S_{sim}\stackrel{}{\to }{S}_{true},when\underset{j=1,2,\dots ,N}{P{C}_{sim\_j}\to P{C}_{SGSP\_j}}$$

(1)

$$\begin{array}{ll}\min & f({\Phi }_{SGSP\_j},{\Phi }_{sim\_j}(\xi ))j=1,2,\dots ,N\\ & \xi =\text{{}{\xi }_{model},{\xi }_{view}{}_{\_j}\text{}}\end{array}$$

(2)

We performed the optimization as expressed in Equation (2). Φ_{SGSP_j} is a feature image after binary and cutting processing of PC_{SGSP_j}. The cutting area is the minimum enclosing rectangle (MER) of the projection-contour of the building. Meanwhile, Φ_sim__j(ξ) is the simulated feature image after binary and cutting processing of PC_{sim_j}(ξ), which need to be generated from the dynamic simulated 3D model, cut by the MER of the projection-contour of building, and resampled to the same size as the corresponding Φ_{SGSP_j}.

Elements of ξ are divided into two parts: (1) parameters of the simulated 3D basic-unit-models ξ_model, such as length, width, and height, which define the structure of buildings; and (2) parameters of the view ξ_{view_j} = {Azimuth_{_j}, Pitch_{_j}}, which are the imaging angle of the j-th image of SGSPs, defined as in Figure 1. If ξ_{view_j} values are known, it can be preset. Moreover, if one or more Azimuth_{_j} and Pitch_{_j} are roughly known, we need to initial values of ξ_{view_j}, and optimize them to find the matching view of the corresponding SGSP for more precise generation of Φ_sim__j(ξ).

The difference between each Φ_{SGSP_j} and the dynamic Φ_sim__j(ξ) need to be compared whenever the parameter-vector ξ changed, and obtain the objective function f(Φ_{SGSP_j}, Φ_sim__j(ξ)). An optimized algorithm is employed to search ξ_best by iteration, and then use ξ_model of ξ_best to reconstruct the 3D shape of the building.

2.1. Parameterized 3D Simulated Models and Projection

To obtain the dynamic Φ_sim__j(ξ), groups of parameterized 3D models are needed. Inspired by [12], some typical parameterized 3D basic-unit-models were designed that could fit most cases of ridge buildings, as shown in Figure 2. Expressed by Equation (3), it is a universal model for these types of buildings. The basic-unit-model could be expanded to fit different applications if necessary.

$$F({\xi }_{model})=F(w,l,{\eta }_1,{\eta }_2,{\eta }_3,{\eta }_4,H_g,H_c).$$

(3)

Commonly, a simple 3D shape can be described using these basic-unit-models. A complex 3D shape needs to be described by a group of basic-unit-models in a certain topological initial state. ξ_model = {ξ_{b_i}}, i = 1, 2, 3, …, K, K is a number of basic-unit-models.

With the parameterized 3D model, we set the entire model ξ_model at the center of the simulated 3D coordinate system. Then, the projection-image of the 3D model of any view could be simulated by Equation (4), using some integrated functions of 3D visualization according to the simulated imaging model V. For aerial images, the perspective projection model is employed, and a generalized model, such as the rational function model (RFM), is used for satellite images. Whenever the parameter ξ is changed, the 3D model on the screen is shown by the 3D visualization functions, and the screenshots are saved as the simulated projection.

$$P{C}_{sim\_j}\left(\xi \right)=V(M({\xi }_{model}),{\xi }_{view\_j}).$$

(4)

2.2. Dynamic Multi-Projection-Contour Approximating by an Artificial Bee Colony Algorithm

In both PC_SGSP and PC_sim, the building areas are set to 1, while other parts are set to 0. We then cut the MER to obtain Φ_{SGSP_j} and Φ_sim__j(ξ). Φ_sim__j(ξ) is then resampled to the same size as Φ_{SGSP_j}, and Equation (5) is employed as an objective function. Similarity S_Ö or normalized similarity Ŝ_Ö can be used to define the similarity between all Φ_{SGSP_j} and Φ_sim__j(ξ) values, as expressed in Equation (6). Then, an artificial bee colony (ABC) algorithm can be employed to resolve the optimization.

The artificial bee colony algorithm groups the bee colony into three roles: employed bees (EB), onlooker bees (OB), and scout bees (SB), which are also the three phases of the algorithm. The main process is as follows:

$$\begin{array}{l}f=-S_{\Phi }({\Phi }_{SGSP\_j},{\Phi }_{sim\_j}(\xi ))\\ \xi =\underset{i=1,2,\dots ,K}{\text{{}{\xi }_{model},{\xi }_{view\_j}\text{}}=\{w_i,l_i,{\eta }_i{}_1,{\eta }_i{}_2,{\eta }_i{}_3,{\eta }_i{}_4,H_i{}_g,H_i{}_c,Azimut{h}_{\_j},Pitc{h}_{\_j}\}}\end{array}$$

(5)

$$S_{\Phi }=\sqrt{{\displaystyle \sum _{j=1}^N{\left(\frac{{\Phi }_{SGSP\_j}\text{\&}{\Phi }_{sim\_j}(\xi )}{{\Phi }_{SGSP\_j}|{\Phi }_{sim\_j}(\xi )}\right)}^2}}\begin{array}{cc}or& {\widehat{S}}_{\Phi }=\frac{1}{N}\sqrt{{\displaystyle \sum _{j=1}^N{\left(\frac{{\Phi }_{SGSP\_j}\text{\&}{\Phi }_{sim\_j}(\xi )}{{\Phi }_{SGSP\_j}|{\Phi }_{sim\_j}(\xi )}\right)}^2}}\end{array}.$$

(6)

(1) Parameter Initialization of the ABC Algorithm

Population number (PN), dimension of ξ (D), triggering threshold of scout bee (TTSB), maximum cycle number (MCN), or ideal fitness threshold (IFT) can be set for termination.

(2) Initialization of Solutions

The solution vector ξ is composed by all parameters to be optimized. Initiate PN/2 solutions ξ_m(n) (m = 1, 2, …, PN/2, n = 1, 2, …, D) randomly according to predefined logical ranges for each element of ξ. PN/2 failure counters FC_m are reset for each solution.

(3) Iteration and Modify all PN/2 Solutions

1)

The EB phase. Each EB corresponds to a solution. Each solution ξ_m(n) is modified by Equation (7), where ξ_k(n) is another solution in the PN/2 solutions, λ_m(n)~U[−1, 1].

$${\overline{\xi }}_m^{}(n)={\xi }_m^{}(n)+{\lambda }_m^{}(n)\cdot ({\xi }_m^{}(n)-{\xi }_k^{}(n)).$$

(7)

To improve the efficiency of the optimization, instead of modifying one element randomly in the original algorithm, all D elements of ξ_m are modified. f(ξ_m) and f(${\overline{\xi }}_m^{}$) are compared, and the better one is reserved to update the solution of this EB. The FC_m will add one if f(ξ_m) > f(${\overline{\xi }}_m^{}$); otherwise, it remains unchanged.

2)

The recruiting probability of each EB is estimated using Equation (8).

$$prob(m)=f\left({\xi }_m\right)/{\displaystyle \sum _{m=1}^{PN/2}f\left({\xi }_m\right)}.$$

(8)

3)

The OB phase: OB optimizes the solution found by the EB it follows. Each OB follows one EB according to recruiting probability, and the solution of that EB is modified. Since better solutions having greater recruiting probability, they can be optimized more times. The modification is the same as the EB phase.

4)

The best solution ξ_best is recorded. All the updated PN/2 solutions are ranked according to their fitness, and ξ_best is recorded. Then, if the algorithm comes to the MCN or f(ξ_best) reaches IFT, it ends. ξ_model of the ξ_best is chosen for further reconstruction.

5)

The SB phase: SB is triggered only if FC_m reaches TTSB, the corresponding solution is abandoned, and a new solution will be randomly generated and the original one replaced.

3. Experiments

In the experiments, the same ABC parameters are used to fairly test the performance of the DMPCA in different SGSP cases. The programs were run in MATLAB 2015, and the computer is with Interl(R) Core(TM) i5-6200 2.3GHz and 8G RAM. Considering the computing ability and time-consuming problem, the main parameters of the ABC algorithm were set as PN = 10, TTSB = 50, and MCN = 100.

3.1. An Analysis of 3D Reconstruction by DMPCA Using Simulated SGSPs

Firstly, we intended to find the relationship between similarity S_Ö and the precision of the reconstruction.

Since imaging angles of the actual remote sensor cannot be precisely controlled, we picked three typical buildings with different 3D shapes, as shown in Figure 3 and

Table 1. Typical buildings (unit: m).

Buildings	l	W	η1	η2	η3	η4	HG	HC
B1	50	30	0	0	0	0	30	0
B2	50	30	15	15	0	0	30	10
B3	50	30	15	15	25	0	30	10
B4_L	50	20	10	10	10	10	20	5
B4_R	50	20	10	10	10	10	20	5
B4_M	40	20	10	10	10	10	20	5

, and simulated their SGSPs with accurate imaging angles, as shown in

Table 2. Views of simulated images (unit: °).

Form of View: (AzimuthSGSP, PitchSGSP)
(0, 30)	(0, 45)	(0, 60)	(0, 75)	(0, 90)
(30, 30)	(30, 45)	(30, 60)	(30, 75)	(30, 90)
(60, 30)	(60, 45)	(60, 60)	(60, 75)	(60, 90)
(90, 30)	(90, 45)	(90, 60)	(90, 75)	(90, 90)
Special:	(150, 45)		(225, 45)

. The resolution of the SGSPs is 1 m/pixel. For convenience, azimuth and pitch angles are defined according to the center of the buildings, as shown in the bottom-right section of Figure 1.

Due to the symmetry of the three buildings, we picked azimuth and pitch angles of 20 regular simulated images (RSI) in 0~90°, and two special simulated images (SSI) are (150°, 45°) and (225°, 45°). Then, we chose one SSI together with any RSI to constitute SGSPs for 3D reconstruction. We compared the error of reconstruction by Equation (9), where NT is the number of testing points, and we compared the actual coordinate (X_n, Y_n, Z_n) with the estimated ones (${\widehat{X}}_n$, ${\widehat{Y}}_n$, ${\widehat{Z}}_n$).

$$PRE=\frac{1}{NT}{\displaystyle \sum _{n=1}^{NT}\sqrt{{\left({\widehat{X}}_n-X_n\right)}^2+{\left({\widehat{Y}}_n-Y_n\right)}^2+{\left({\widehat{Z}}_n-Z_n\right)}^2}}$$

(9)

We first tested the minimum requirements of similarity S_Ö for different precisions of 3D reconstruction by our DMPCA method. It can be seen in Figure 4a,b that similarity S_Ö, defined by the feature-image Φ, can be an effective restriction of the 3D shape, and higher precision requires a larger S_Ö from the trend for different SGSP combinations. Nevertheless, in the same precision, the requirements of S_Ö are different for different buildings, also for different cases of SGSPs.

Two views were used in the experiments of B1~B3, and the three buildings are entirely symmetrical. Furthermore, B4 was used, which is not perfectly symmetrical but was a complex building. Then, the performance of DMPCA on B4 was tested in different combination cases of three different views;

We focused on the effects of the azimuth spans of DMPCA, and the pitch angles of input images were fixed at 45°. Eleven views of B4 were used, as shown in Figure 5. First, the cases of equidistant azimuth spans were considered. The initial azimuth was set to 0°, and the azimuth spans between two adjoined images were set to five cases (0°, 30°, 45°, 60°, 90°, and 120°), so the combinations of azimuths of the five cases were as follows: 0-30-60, 0-45-90, 0-60-120, 0-90-180, and 0-120-240. The results after 100 iterations are shown in

Table 3. Performance of DMPCA on different cases of equidistant azimuth spans.

Azimuth Spans	Similarity	Precision (m)
30°	0.815	1.73
45°	0.822	1.66
60°	0.841	1.56
90°	0.872	1.37
120°	0.923	1.07

. It could be found that, in the case of equidistant azimuth spans, a larger azimuth span produces better precision.

Moreover, also in equidistant azimuth spans, two-view cases and four-view cases were tested, similar to the above three-view cases (note that, when using four views of the 120° azimuth span, two of the four were entirely overlapped; this input case is the same as the case using three views of the 120°, but the one redundancy image deduced the optimizing process and, compared with the case using three views, obtained a worse result). It was illustrated that three views are the better choice in equidistant azimuth spans, as shown in Figure 6.

According to a number of experiments, it was found the initial azimuth also has certain effects on the results, since the results of a 0° initial azimuth is no better than other cases of initial azimuth. The initial azimuth was then set to 60°, and the cases of non-equidistant azimuth spans were tested. The views and results are shown in

Table 4. The performance of DMPCA on different cases of non-equidistant azimuth spans.

The Azimuths	Similarity	Precision (m)
60-150-180	0.930	0.8400
60-150-210	0.961	0.4800
60-150-240	0.975	0.3000
60-150-270	0.976	0.2880
60-150-300	0.988	0.1440
60-180-210	0.948	0.526
60-180-240	0.969	0.512
60-180-270	0.973	0.5115
60-180-300	0.985	0.5085

, Figure 7 and Figure 8. After 100 iterations, the 60-150-300 case obtained the highest value of similarity and precision. Moreover, it can be found that similarity and precision do not increase simultaneously.

It can be concluded from above that the complementarity of the azimuth is the most significant factor of the DMPCA method.

3.2. 3D Reconstruction by DMPCA Using Actual Remote Sensing SGSPs

We tried to reconstruct the main building of the Harbin institute of technology (HIT). As shown in Figure 9, we collected a satellite image (SI) from a vendor and two aerial images (AI1, AI2) by our unmanned aerial vehicle.

It could be found that aerial images are oblique, while satellite images are nadir-looking. Moreover, compared to AI2, SI and AI1 have similar azimuths. We combined them two by two and obtained three groups of SGSPs (SGSP1 = SI & AI1, SGSP2 = SI & AI2, SGSP3 = AI1 & AI2). Then, we reconstructed the building using the three groups of SGSPs, respectively.

The projection-contours of the building were extracted from each SGSPs using the image matting methods [13] and some manual correction. Since the building has a complex 3D shape, we initialized it into 27 units (ξ_model = {ξ_{b_i}}, i = 1, 2, …, 27) as Figure 9e, including 25 outer-units and 2 inner-units. For each basic unit, the horizontal parameters were resolved from the SI referring to the building extraction methods in [14]. Then, the height parameters of 27 basic-unit-models and view parameters were collected to compose ξ. Note that we were approximating the projection-contour of the entire model of the building, not each single unit. Therefore, the objective function f in Equation (5) needs no changes.

According to the design diagram of this building, we collected the true-value of the top of the building. Each reconstructed unit was sampled into 0.1 m interval 3D points-cloud, and we selected all the sampled points of the roof of building as testing points, to test the precision by Equation (9). The results are shown in

Table 5. Reconstruction results obtained by different SGSPs. “Precision” is the mean PRE of the 27 basic-unit-models. Figures in the last column show the error of 3D reconstruction; the x-axis is the serial number of units, and the y-axis is PRE (unit: m).

SGSPs	Precision	Visualization of the 3D Reconstruction	Error of 3D Reconstruction
SGSP1	3.1 (m)
SGSP2	2.1 (m)
SGSP3	1.0 (m)

.

The results revealed that, overall, 3D reconstruction of the building could be achieved by any of the three SGSPs using our DMPCA. A larger S_Ö produces better results, which support our hypothesis as shown in Equation (1). In fact, a larger S_Ö is sufficient but not necessary for better reconstruction, since our simulated perspective projection model cannot fit the actual imaging model perfectly. The aerial SGSP3 with higher resolution produces better precision than SGSP1 and SGSP2. Particularly, although the precision of several units obtained by SGSP2 are poorer than that obtained by SGSP1, these units are in small proportion of the entire building, so SGSP2 obtained better mean precision than SGSP1. The only difference between SGSP1 and SGSP2 is the azimuth of AI1 and AI2. We think that this made SGSP2 a better complementarity with SI. This indicated that the precision of 3D reconstruction by our DMPCA framework mainly depends on the resolution and complementarity of the imaging angle of the SGSPs. Moreover, the precision of each unit, obtained by the respective SGSPs, is different. The inner-units could not be effectively restricted by the projection-contour, since there is only a little piece of part or none of the inner-units in the projection-contour. Thus, the precision of the inner-units cannot be guaranteed, which mostly depends on the pitch angle of the view. However, most of the outer-units have an ideal precision according to their inputting SGSPs.

4. Discussion and Conclusions

A dynamic multi-projection-contour approximating (DMPCA) framework was proposed in this paper. Using super-generalized stereo-pairs (SGSPs), the reconstruction of the 3D shape of a building was achieved, while traditional frameworks failed. In this framework, we resolved an optimization to find the best parameters of a certain group with a simulated 3D model, which minimizes differences between the projection-contour of a building in the actual optical images and that of the simulated 3D model. According to experiments, we found that the proposed similarity can well restrict the shape of the building. However, the similarity and precision do not increase simultaneously. There are two possible reasons: (1) our projection model is not absolutely consistent with the actual imaging model; (2) the optimizing algorithm did not find the best solution of the parameters. Moreover, the effect of azimuth spans on DMPCA were analyzed. In the equidistant azimuth spans cases, we found that a greater number of views will not produce a better result, and three views could be a suitable choice. In the non-equidistant azimuth spans cases, it was proved that the azimuth spans, which has better complementarity, can improve the performance of DMPCA, since they have better restriction of the shape. In the test of actual SGSPs, the precision of 3D reconstruction by our DMPCA mainly depends on the resolution and complementarity of the imaging angle of the SGSPs. A higher resolution and better complementarity of azimuth produce better precision.

Furthermore, to improve this work, a basic-unit-model could be better defined for greater application, and the optimized algorithm could be improved to enhance the performance.