Planar Reconstruction of Indoor Scenes from Sparse Views and Relative Camera Poses

Abstract

Planar reconstruction detects planar segments and deduces their 3D planar parameters (normals and offsets) from the input image; this has significant potential in the fields of digital preservation of cultural heritage, architectural design, robot navigation, intelligent transportation, and security monitoring. Existing methods mainly employ multiple-view images with limited overlap for reconstruction but lack the utilization of the relative position and rotation information between the images. To fill this gap, this paper uses two views and their relative camera pose to reconstruct indoor scene planar surfaces. Firstly, we detect plane segments with their 3D planar parameters and appearance embedding features using PlaneRCNN. Then, we transform the plane segments into a global coordinate frame using the relative camera transformation and find matched planes using the assignment algorithm. Finally, matched planes are merged by tackling a nonlinear optimization problem with a trust-region reflective minimizer. An experiment on the Matterport3D dataset demonstrates that the proposed method achieves 40.67% average precision of plane reconstruction, which is an improvement of roughly 3% over Sparse Planes, and it improves the IPAA-80 metric by 10% to 65.7%. This study can provide methodological support for 3D sensing and scene reconstruction in sparse view contexts.

1. Introduction

Planar surfaces play a crucial role in three dimension (3D) sensing technology, which is used in a variety of applications such as robot navigation [1,2], 3D reconstruction [3,4], and scene understanding [5,6]. More specifically, Dong et al. [7] locate autonomous vehicles in underground mines by combining image recognition technology and data from other sensors. Xie et al. [8] extract planar primitives from point clouds and reconstruct 3D building models based on topological relationships. Li et al. [9] decomposite 3D space by partitioning different buildings based on planar primitives. As planar surfaces provide a more compact representation than mesh-based non-planar models, they are more computationally efficient [10]. Thus, planar reconstruction has been a focus of computer vision research for years: for example, piece-wise planar reconstruction from point clouds [3] and Manhattan-world reconstruction from multiple images [11].

While traditional methods can produce multi-view geometry [12,13], they rely on correspondence for reconstruction and are limited to the part of the scene that has direct overlaps across images. In recent years, learning-based single-view 3D tools have improved a lot. They can reconstruct 3D planar surfaces from a single input image [14,15], which has inspired more scene reconstruction methods under sparse view settings [16]. However, neither of those methods takes advantage of the relative camera pose when performing reconstruction.

To address this problem, this paper proposes a new method to produce a unified scene reconstruction from two images and their relative camera pose. With the additional input of the relative camera pose, the proposed method can overcome some of those limitations. Following existing work in this area [16,17], this study produces planar reconstruction using a three-step approach: detecting planes in each image, matching planes across images, and merging planes. Firstly, it detects planes in both images and predicts the plane mask, plane parameters (normal and offset), and appearance embedding features for each plane by a PlaneRCNN-based architecture [15]. Secondly, it matches planes between images by considering appearance and geometric features, including plane parameters projected by the relative camera pose, to find planes that appear in both images. Finally, matched planes are merged by optimizing their plane parameters to produce a unified scene reconstruction.

The major contributions of this paper are as follows:

1.

We present a novel framework for 3D planar reconstruction from two sparse views and their relative camera pose based on a merging framework that combines PlaneRCNN for plane detection and a trust region reflective for optimization.

2.

We present a method for matching cross-view plane segments that utilizes appearance embedding and geometric features.

3.

We develop an efficient solution for reconstructing 3D planar representations of indoor scenes from sparse views and relative camera poses that achieves 40.67% average precision for reconstructing plane segments and 65.7% IPAA-80 for cross-view plane matching accuracy.

In the subsequent sections, we review previous studies (Section 2), introduce our method in detail (Section 3), and present the experimental details along with the results of comparisons (Section 4). Finally, we give our conclusions in Section 5.

2. Related Work

The proposed method for planar reconstruction in 3D space draws upon some well-suited tasks from computer vision topics, ranging from single-view or multi-view 3D reconstruction of RGB images to correspondence estimation in multi-view scenes.

Reconstruction from a single image: This is a well-known computer vision task with established methods that aims to infer 3D structures from 2D images and includes depth maps [18,19], normal maps [20,21], point clouds [22,23], meshes [24,25], voxels [26,27], and planes [28,29]. The proposed method achieves 3D reconstruction by planes, as they are often good approximations of indoor scenes [30]. PlaneNet, proposed by Liu et al. [28], solves piece-wise planar depth map reconstruction problems from images of indoor scenes with an end-to-end ML (machine learning)-based framework. While it is capable of inferring plane parameters, there is a lot of room for improvement in terms of planar segmentation. PlaneRecover, proposed by Yang et al. [29], uses an unsupervised learning approach for outdoor scenes. However, both of these methods can only cover segmentation problems with a fixed number of planar regions (10 in PlaneNet and 5 in PlaneRecover), limiting their reconstructions’ expressiveness. Yu et al. [31] segment plane instances and recover 3D plane parameters from an image via a two-stage method based on associative embedding that maps pixels to an embedding space and groups them in planar regions, and their method can detect an arbitrary number of planes, but it may fail when subjected to adjacent planes with similar textures because of a lack of semantic information. PlaneRCNN, proposed by Liu et al. [15], employs a variant of Mask R-CNN [32] to detect planes with their plane parameters in a single step; then, it jointly refines all the segmentation masks using a refinement network. Even though it is trained on indoor scenes, PlaneRCNN is able to reconstruct most planes from unseen datasets regardless of their shapes and textures. The detection backbone in the proposed method is based on the PlaneRCNN architecture. In contrast to PlaneRCNN, the proposed method considers appearance and position across multiple views and can produce a unified scene reconstruction.

Multiview 3D reconstruction: While traditional methods can produce multi-view geometry [12,13], they rely on correspondence for reconstruction and are limited to the part of the scene that has direct overlaps across images. In particular, they rely on homographies or triangulation as a cue and usually fail when input images have little overlap. Other methods [33,34] utilize a plane-sweep algorithm to establish matching confidence and subsequently refine the disparity of the reference frustum based on this confidence. While some methods have proven successful under optimal Lambertian conditions [35], they tend to suffer from poor generalizability to other settings. This often leads to incomplete or incorrect reconstructions due to low surface textures, occlusion of objects, or transparency. ML-based single-view 3D tools [14,15] can produce reconstructions from single input images. Still, it is challenging to merge reconstruction between views to reconstruct one unified scene from multiple images: figuring out the position of an extracted 3D object requires joint reasoning appearance features, geometry relationship, and translation between camera poses. Xi et al. [36] enhance the capability of piece-wise planar reconstruction by multi-view regularization during the training phase, improving the consistency among multiple views by making the feature embedding more robust. PlaneMVS, proposed by Liu et al. [37], decouples plane reconstruction into a semantic plane detection branch and a plane MVS (multi-view stereo) branch and takes advantage of multi-view geometry. Its semantic plane detection branch is also based on PlaneRCNN [15].

Correspondence estimation: Estimating correspondences across images is vital for producing 3D reconstructions from multiple images [38,39,40]. The wide baseline stereo (WBS) matching problem formulated by Pritchett et al. [12] has received significant attention for many years [41,42]. Existing algorithms [43] and frameworks [44] have successfully matched views of planar objects with orientation differences of up to ${160}^{\circ }$. However, viewpoint orientation is not the only factor that influences the complexity of establishing geometric correspondence between images. The WxBS (wide “properties” baseline stereo) problem introduced by Mishkin et al. [45] considers more properties like illumination, geometry, appearance, and sensor type and solves the problem by repeating the matching steps on multiple local feature descriptors. There are also SLAM-based (simultaneous localization and mapping) methods like [46,47], which, unlike stereo-based ones, use bundle adjustment to estimate 3D correspondence and often rely on a sequence of RGB-D images as input. Yi et al. trained an end-to-end network by embedding global information in each data point to filter out putative sparse matches between two distinct views. Choy et al. [40] proposed a framework for pairwise 3D model registration by predicting correspondence using a convolutional network and optimizing it later. This paper takes inspiration from these methods to produce cross-image correspondence between plane segments.

Reconstruction from sparse views: Under sparse view settings, many existing methods can estimate the camera pose but do not produce 3D scene reconstruction [48,49]; other methods that do have this capability come with limitations as well. DSNeRF, proposed by Deng et al. [50], overcomes the problem of having an insufficient number of input views by training the radiance fields with readily available depth supervision. SparseNeRF, proposed by Wang et al. [51], complements the lack of 3D information in sparse views by exploiting depth priors from inaccurate depth observations produced by pre-trained depth models. However, NeRF (neural radiance field)-based methods cannot produce full scene reconstruction and only generate novel views from certain angles. In contrast, Qian et al. [16] combined multiple networks to produce a volumetric reconstruction of an indoor scene, but this requires the dataset to have watertight ground-truth meshes and only works under synthetic environments wherein only a few objects are in the scene. Jin et al. [17] predicted planes from two views using an extended PlaneRCNN [15] architecture and jointly estimated plane correspondences with a hand-designed optimization. Agarwala et al. [52] also employed the same plane prediction architecture but used a transformer to predict the plane correspondence. However, neither of these takes advantage of the relative camera pose between images. The proposed method augments plane features with the relative camera pose to improve the accuracy of the reconstructed scene further.

3. Methodology

The proposed method, shown in Figure 1, aims to jointly process two images with their relative camera pose and produce a unified planar reconstruction of the scene where those images were taken. This process requires two pieces of key information: the mask of the planes in the input images to fill in their textures and the correspondence among planes in each view to merge the same plane that appears in both images. This can be done with a network that predicts plane parameters and appearance embeddings, followed by a matching module that considers both the relative camera pose and appearance, and finally, the merging of matched planes via joint optimization to produce a unified scene reconstruction. The system is built upon Detectron2 [53] and PyTorch 1.17.

3.1. Detecting Planes in Each Image

The plane detection module is built on top of a single-view plane prediction backbone from [17], which is an improved version of PlaneRCNN [15]. This module extracts planes for each image and predicts parameters for each plane. As shown in Figure 2, given the ith image, the detected plane is indexed by j and has a mask segment ${\mathcal{S}}_{i,j}$, plane parameters ${\pi }_{i,j}\in {\mathbb{R}}^4$, and appearance embedding ${\mathbf{e}}_{i,j}\in {\mathbb{R}}^{128}$. The plane parameter ${\pi }_{i,j}$ consists of a unit vector normal ${\mathbf{n}}_{i,j}$ and an offset $o_{i,j}$ that defines a plane in 3D space by equation ${\mathbf{n}}_{i,j}^T[x,y,z]-o_{i,j}=0$.

Feature extraction: The module uses ResNet-50-FPN [54] pre-trained on COCO [55] as the backbone feature extraction layer. Given an input image, this layer produces feature maps with resolutions of $2\times 2,4\times 4,8\times 8,16\times 16,$ and $32\times 32$.

Plane detection: Following PlaneRCNN, the plane extraction module is built on Mask R-CNN [32]. This module uses a region proposal network (RPN) [56] to generate a set of candidate RoIs (regions of interest) and filter out invalid candidates that cannot fit any planes in their bounding boxes using RoI align. The mask branch uses a fully convolutional network (FCN) [57] to produce semantic segmentation mask $\mathcal{S}$ for each plane instance.

Instead of predicting object categories, the normal branch predicts one of seven anchor normals from K-means clustering in 10,000 randomly sampled training images [15]. Then, it regresses the residual 3D vector to refine the normal prediction.

Additionally, the backbone features are also piped into a decoder to estimate the depth map of the entire input image with the same resolution. Starting with the smallest $2\times 2$ feature map, the decoder upsamples it and concatenates the upsampled feature map with the next-larger feature map. After all the feature maps have been processed, a $3\times 3$ convolution regresses the depth map and upsamples it to the image solution with bilinear interpolation.

Once the normal $\mathbf{n}$ and depth map have been determined, the plane offset o can be estimated by summing over all the pixels in the image:

$$o=\frac{{\sum }_i{b}_i\left({\mathbf{n}}^{\top }\left(d_i{K}^{-1}{\mathbf{x}}_i\right)\right)}{{\sum }_i{b}_i}$$

(1)

where K is the $3\times 3$ camera-intrinsic parameter matrix, ${\mathbf{x}}_i$ is the $i$th pixel coordinate in a homogeneous representation, $d_i$ is the predicted depth of the pixel, and indicator variable $b_i$ becomes 1 if the pixel belongs to the plane.

Appearance embedding: The appearance embedding is produced by a network trained on pairs of images with triplet loss. Following [17], cross-view plane correspondence triplets are defined as ${\mathbf{e}}_a,{\mathbf{e}}_p^{\prime },{\mathbf{e}}_n^{\prime }$, where anchor ${\mathbf{e}}_a$ corresponds to the positive match ${\mathbf{e}}_p^{\prime }$ but not the negative match ${\mathbf{e}}_n^{\prime }$. The standard triplet loss [58] is defined as:

$$max\left(\right||{\mathbf{e}}_a-{\mathbf{e}}_n^{\prime }|{|}_2-\left|\right|{\mathbf{e}}_a-{\mathbf{e}}_p^{\prime }|{|}_2+\alpha ,0)$$

(2)

The anchor and positive must be closer than the anchor and negative by a margin of $\alpha =0.2$; otherwise, it will give a loss. When training the network, online triplet mining is used to generate triplets of data points dynamically. As a result, the appearance embedding vector ${\mathbf{e}}_i$ can be used for cross-view matching: given images i and $i^{\prime }$ of the same scene, whenever plane $j^{\prime }$ matches plane j, the distance between their embedding vectors $\left|\right|{\mathbf{e}}_{i,j}-{\mathbf{e}}_{i^{\prime },j^{\prime }}\left|\right|$ should be small.

3.2. Matching Planes by Appearance and Relative Camera Pose

The plane matching module is designed to find every plane that appears in each input image by predicted plane parameters and appearance embedding. As input, there is a set of m planes for Image 1 $\{{\mathcal{S}}_{1,j},{\pi }_{1,j},{\mathbf{e}}_{1,j}\}$ and a set of n planes for Image 2 $\{{\mathcal{S}}_{2,j},{\pi }_{2,j},{\mathbf{e}}_{2,j}\}$ as well as the relative camera pose represented by a camera-to-camera transform: the 3D translation vector $\mathbf{t}$ and translation matrix $\mathbf{R}$ produced by the rotation. The translation vector $\mathbf{t}=(d_x,d_y,d_z)$ contains the translation distances of the cameras on the $X,Y,Z$ axes, respectively. The rotation between cameras can be expressed as quaternion $q=a+bi+cj+dk$, where $i^2=j^2=k^2=ijk=-1$. The translation matrix $\mathbf{R}$ an be calculated from the quaternion q to improve the efficiency of subsequent computations:

$$\mathbf{R}=\left[\begin{array}{ccccc}1-2c^2-2d^2& & 2bc-2ad& & 2ac+2bd\\ 2bc+2ad& & 1-2b^2-2d^2& & 2cd-2ab\\ 2bd-2ac& & 2ab+2cd& & 1-2b^2-2c^2\end{array}\right]$$

(3)

The first step is to transform the normal ${\mathbf{n}}_{2,j}$ and the offset $o_{2,j}$ of the plane parameters ${\pi }_{2,j}$ from the second image by $\mathbf{R}$ and $\mathbf{t}$ to fit them in the same coordinates of the first image, resulting in a new set of plane segments ${\mathit{\pi }}_{2,j}^{\prime }=[{\mathbf{n}}_{2,j}^{\prime },o_{2,j}^{\prime }]$:

$$\begin{array}{cc}\hfill \mathbf{P}& =\left(1+\frac{\mathbf{t}·\mathbf{R}(o_{2,j}·{\mathbf{n}}_{\mathbf{2},\mathbf{j}})}{\left|\right|\mathbf{R}(o_{2,j}·{\mathbf{n}}_{\mathbf{2},\mathbf{j}}){\left|\right|}^2}\right)\mathbf{R}(o_{2,j}·{\mathbf{n}}_{\mathbf{2},\mathbf{j}})\hfill \\ \hfill {\mathbf{n}}_{2,j}^{\prime }& =\mathbf{P}/\left|\right|\mathbf{P}\left|\right|\hfill \\ \hfill o_{2,j}^{\prime }& =\left|\right|\mathbf{P}\left|\right|\hfill \end{array}$$

(4)

The next step is to define a cross-view plane correspondence matrix $\mathbf{C}\in {\{0,1\}}^{m\times n}$, for which ${\mathbf{C}}_{i,j}$ becomes 1 only when the ith plane in Image 1 corresponds to the jth plane in Image 2. To find the closet matches, the network should maintain an $m\times n$ cost matrix $\mathbf{K}$ that encodes the quality of plane matching $\mathbf{C}$:

$${\mathbf{K}}_{i,j}={\mathbf{K}}_{i,j}^{\mathrm{normal}}+{\mathbf{K}}_{i,j}^{\mathrm{offset}}+{\mathbf{K}}_{i,j}^{\mathrm{embedding}}$$

(5)

where ${\mathbf{K}}_{i,j}^{\mathrm{normal}},{\mathbf{K}}_{i,j}^{\mathrm{offset}},{\mathbf{K}}_{i,j}^{\mathrm{offset}}$ encodes the costs between the plane normal vector, plane offset, and appearance embedding, respectively.

The similarity of plane normals can be encoded using cosine similarity: given two vectors, in this case, plane normal ${\mathbf{n}}_{1,i}$ and ${\mathbf{n}}_{2,j}$, the cosine similarity can be calculated as

$$\mathrm{similarity}=cos\langle {\mathbf{n}}_{1,i},{\mathbf{n}}_{2,j}\rangle =\frac{{\mathbf{n}}_{1,i}·{\mathbf{n}}_{2,j}}{\left|\right|{\mathbf{n}}_{1,i}\left|\right|\left|\right|{\mathbf{n}}_{2,j}\left|\right|}$$

(6)

Since the plane normals have already been normalized, the modulus of these vectors is always 1. The cosine similarity value ranges from −1, meaning exactly opposite, to 1, meaning exactly the same. The cost between two identical normals should be 0, so the actual cost can be calculated by subtracting the similarity from 1:

$${\mathbf{K}}_{i,j}^{\mathrm{normal}}=1-{\mathbf{n}}_{1,i}·{\mathbf{n}}_{2,j}$$

(7)

Because the plane offset value is just a 1D number, the offset cost can be encoded as the difference between those two values. And the embedding cost can also be encoded as the vector distance in the embedding space:

$${\mathbf{K}}_{i,j}^{\mathrm{offset}}=|o_{1,i}-o_{2,j}^{\prime }|$$

(8)

$${\mathbf{K}}_{i,j}^{\mathrm{embedding}}=\left|\right|{\mathbf{e}}_{1,i}-{\mathbf{e}}_{2,j}^{\prime }\left|\right|$$

(9)

Finally, the cost $\mathbf{K}$ of cross-view plane correspondence can be encoded by combining all of those factors with a certain weight:

$${\mathbf{K}}_{i,j}=k_n(1-{\mathbf{n}}_{1,i}·{\mathbf{n}}_{2,j})+k_o|o_{1,i}-o_{2,j}^{\prime }|+k_e\left|\right|{\mathbf{e}}_{1,i}-{\mathbf{e}}_{2,j}^{\prime }\left|\right|$$

(10)

where $k_n,k_o,k_e$ are weight parameters for the normal, offset, and embedding, respectively, which are fit using a randomized search on the validation set.

According to the cost matrix, a cost value of 0 indicates a perfect match between planes, with the same normal and offset and an exact match in the embedding space. With the cost matrix determined, this matching problem can be efficiently solved using the Hungarian algorithm [59]. However, in real-world scene scenarios, not every plane in one image would match a plane in another image, so there needs to be a threshold to reject matches with large distance errors.

3.3. Merging Matched Planes

Now that the network has detected planes in both images and found matched planes across images, the final step is to align and merge matched segments into one coherent plane by refining predictions from the deep networks.

To align plane segments from input images with different view perspectives, the bounding box $b_i(i\in \{1,2,3,4\})$ of the segment is first projected to 3D space, then to the normalized global frame where its normal is parallel to the viewing perspective, as demonstrated in Figure 3.

Given a plane with parameters normal $\mathbf{n}$ and offset o, point $(x,y)$ on the plane can be projected into 3D space as point $(x^{\prime },y^{\prime },z^{\prime })$:

$$d=\frac{o}{\mathbf{n}·\left(K^{-1}{[x,y,1]}^{⊺}\right)}$$

(11)

$${[x^{\prime },y^{\prime },z^{\prime }]}^{⊺}=d·\left(K^{-1}{[x,y,1]}^{⊺}\right)$$

(12)

where d is the estimated depth of the point based on the plane parameters, and K is the $3\times 3$ camera intrinsic matrix. Then, the bounding box of the segment in the global frame can be calculated with:

$$\begin{array}{cc}\hfill \widehat{x}& =\frac{b_2^{\prime }-b_1^{\prime }}{\left|\right|b_2^{\prime }-b_1^{\prime }\left|\right|}\hfill \\ \hfill \widehat{y}& =\mathbf{n}\widehat{x}\hfill \\ \hfill b_i^{″}& =[(b_i^{\prime }-b_1^{\prime })\widehat{x},(b_i^{\prime }-b_1^{\prime })\widehat{y}],i\in \{1,2,3,4\}\hfill \end{array}$$

(13)

Afterward, a $3\times 3$ matrix $\mathbf{P}$ of the perspective transform that projects the pixels of the plane in the image to the global frame can be found:

$$\left[\begin{array}{c}{t}_i{b}_i^{″⊺}\\ t_i\end{array}\right]=\mathbf{P}·\left[\begin{array}{c}{b}_1^{⊺}\\ 1\end{array}\right],i\in \{1,2,3,4\}$$

(14)

All matched plane segments are projected to the global frame, and key points can be extracted from those projected pixels using the SIFT (scale-invariant feature transform) [60] feature. To filter out incorrect key points, a random affine transformation is applied to each pair of projected planes: collecting another set of points and transforming those points to the global frame. This process must be repeated several times, followed by a RANSAC (random sample consensus) to select the most likely key points, assuming that those features are not affected by affine transformation [60].

Since plane parameters produced by deep networks are likely to have errors between ground-truth values, and parameters of matched planes between images can also be misaligned due to said errors, it is necessary to minimize these errors before the network can produce a unified scene reconstruction.

The proposed method accomplishes this by optimizing over plane parameters ${\widehat{\mathit{\pi }}}_{1,i},{\widehat{\mathit{\pi }}}_{2,j}^{\prime }$ to minimize the geometric distance between the matched planes and the pixel offset error:

$$\underset{{\widehat{\mathit{\pi }}}_{1,i},{\widehat{\mathit{\pi }}}_{2,j}^{\prime }}{\arg \min }\sum _{i,j}{\mathbf{C}}_{i,j}\left(\right||{\widehat{\mathit{\pi }}}_{1,i}-{\widehat{\mathit{\pi }}}_{2,j}^{\prime }\left|\right|+D_{\mathrm{k}}({\widehat{\mathit{\pi }}}_{1,i},{\widehat{\mathit{\pi }}}_{2,j}^{\prime }))$$

(15)

where $D_{\mathrm{k}}$ measures the Euclidean distance of key points between matched planes. The pixels of those key points are back-projected to 3D to calculate point-wise Euclidean distances. The optimization uses ${\mathit{\pi }}_{1,i},{\mathit{\pi }}_{2,j}^{\prime }$ as initial variables and solves this non-linear problem with the trust region reflective algorithm. This algorithm is particularly suitable for large, sparse problems with bounds. The loss uses a standard least-squares function, and the solver for trust region subproblems is chosen based on the type of Jacobian returned in the first iteration. The tolerance for termination by the change in the cost function, variables, and gradient norm are all set to ${10}^{-8}$.

Once plane parameters are determined, it is finally time to merge matched planes across images. Masks of plane segments can be merged by aligning key points in the global frame, offsets are merged by averaging, and normals are merged by maximizing ${\sum }_j{\left({\widehat{\mathbf{n}}}_j^{⊺}{\mathbf{n}}_{i,j}\right)}^2$.

4. Experiments

The proposed method is evaluated by both qualitative and quantitative results for full-scene reconstruction and plane-matching accuracy. The evaluation is processed on a Dell PowerEdge T640 tower server with an Intel Xeon 5218R CPU and Nvidia GeForce RTX 3080Ti GPU and running Ubuntu 18.04.6 LTS. We implemented the proposed method with Python 3.7.16 and PyTorch 1.13.1, which runs on CUDA 11.7. The non-linear optimization algorithm is implemented with the least_squares optimization method from SciPy 1.7.3.

4.1. Dataset Description

For a fair comparison of reconstruction quality, the exact dataset from [17] is used. The dataset contains rendered views of real-world scenes from the Matterport3D [61] dataset using AI Habitat [62]. AI Habitat is a simulation platform that not only enables the rendering of realistic images but also provides ground truth depth information to be used in the evaluation process. There are 31,392 image pairs in the training set, 4707 image pairs in the validation set, and 7996 image pairs in the test set. The generated views are widely separated: about 21% of the views overlap with ${53}^{\circ }$ relative rotation and $2.3$ m relative translation. Figure 4 shows some random examples of image pairs in the dataset and their overlapping regions.

4.2. Scene Reconstruction Quality

The proposed method is capable of producing reconstruction of real-world scenes represented by 3D plane segments in the global coordinate frame. To begin with, the experiment compares the quality of the reconstructions with various baselines from [17].

Metrics: The scene reconstruction problem can be treated like a detection problem following other approaches that produce individual components for objects [16,63,64,65,66], and we evaluate the quality of reconstruction using the average precision (AP). Given a pair of input images with a relative pose and a set of ground-truth plane segments, the experiment evaluates how well the planes are detected and reconstructed. The metric for full scene reconstruction comes from [17] and counts each reconstructed plane as true positive whenever: (i) mask IoU (intersection-over-union) $\ge 0.5$, (ii) plane normal angle $\le {30}^{\circ }$, and (iii) offset distance $\le 1$ m.

Baselines: The experiment compares the proposed method with full systems like Sparse Planes [17] and PlaneFormers [52]. They both can produce scene reconstruction in the form of plane segments from image pairs but do not require the relative camera pose as input. Some results of the combination of existing systems from [17,52] are also included.

Qualitative Results: Figure 5, Figure 6, Figure 7, Figure 8 show several examples of full scene reconstruction results from two images in the Matterport3D test set. Detected plane segments are annotated with circles and colored differently, with lines indicating matching results. Prediction and ground-truth models are shown in two distinct views to contain all reconstructed plane segments in the same scene. The proposed method detects planes from two views, merges corresponding ones, and produces a unified scene reconstruction.

Comparison between the proposed method and Sparse Planes [17] is presented in Figure 9, Figure 10, Figure 11, Figure 12, Figure 13, Figure 14, Figure 15, Figure 16. As shown in Figure 9, Sparse Planes fails to predict the continuity of the walls and leaves larger holes when reconstructing the wall, while the proposed method manages to construct a coherent indoor structure. Figure 10 demonstrates that Sparse Planes sometimes fails to predict the relative positions of planes, resulting in poor reconstruction quality. The reason may be that these images have a rather large relative camera rotation angle. In Figure 11, Sparse Planes misses some smaller planar segments among larger ones and incorrectly splits the chair next to the wall. Under larger view angles like in Figure 12, the proposed method reconstructs the wall and the floor as a connected corner, but Sparse Planes misplaces the floor far from the wall. For Figure 13, Sparse Planes produces a cluster of disconnected small pieces from a narrow segment of a far-away wall, while the proposed method constructs the wall successfully. Figure 14 shows a situation where, from different viewpoints, different objects can be seen through the door. The proposed method correctly reconstructs the planes behind the door, while Sparse Planes wrongly covers the door with another plane segment next to it. Further, Figure 15 shows a rather complicated scene in which the two images have little overlap and many pieces of furniture are placed next to the wall. Sparse Planes does reconstruct the larger wall with paint on it but fails with the floor. The proposed method manages to connect the floor between those two images and produces a more complete scene. Last but not least, Figure 16 shows that the proposed method can predict the relative positions of smaller plane segments more precisely, while Sparse Planes sometimes leaves gaps between connected planes.

Quantitative Results As shown in

Table 1. AP (average precision), which treats plane reconstruction as plane detection in 3D space with a different definition of true positive. All: mask IoU $\ge 0.5$; plane normal angle $\le {30}^{\circ }$; offset distance $\le 1$ m. -Offset and -Normal: ignores the offset and normal condition.

Methods	All	-Offset	-Normal
Appearance Embedding Only	33.04	39.78	36.85
Associative3D [16] Optimization	33.01	39.43	35.76
Sparse Planes [17]	36.02	42.01	39.04
PlaneFormers [52]	37.62	43.19	40.36
No Optimizing Plane Parameters	39.78	45.32	41.23
Proposed	40.67	45.69	41.85

, the proposed method outperforms previous approaches with the help of relative camera pose input, even without optimizing plane parameters. Although Sparse Planes [17] does not require the relative camera pose as an input, it needs to select a camera pose from a set of 32 predefined ones, which, in turn, comes from k-means clustering of the training set; therefore, it can be inaccurate when the real-world camera pose does not fit in said clusters. PlaneFormers [52] can also produce a probability distribution over a predefined set of camera transformations then apply a transformer to the plane parameters to perform 3D reasoning, which yields better results but still fails for image pairs with uncommon camera poses. Unlike previous methods, the proposed method takes advantage of the relative camera pose to perform plane matching across images, which makes the reconstruction of relative positions between planes more robust and, therefore, yields higher quality 3D scene reconstruction results.

Additionally, the average time consumption to process an input image pair is measured in

Table 2. Average time consumption (in seconds) to process an image pair in the test dataset with an average of 7 planes per image and 4 matching planes across images.

Method	Sparse Planes [17]	PlaneFormers [52]	Proposed
Time Consumption (Seconds)	8.10	5.10	6.37

. For a fair comparison of processing speed, only established methods are included. The test split in the Matterport3D dataset contains seven planes per image and four matching planes across images on average. As previously noted above, Sparse Planes cannot take advantage of the relative camera pose information and needs to select a camera pose from a predefined set; it takes the longest time to process the image and produce a 3D reconstruction. Although PlaneFormers manages to consume less time to generate higher quality reconstruction than Sparse Planes, the proposed method can produce even higher quality results while only increasing the average time consumption by about 1 s. Moreover, PlaneFormers needs two separate models for plane detection and 3D reasoning: it requires more memory to be able to run the algorithm at all.

4.3. Plane Matching Accuracy

One challenging point for producing a single unified scene reconstruction from two images is ensuring that every plane that appears in both images is reconstructed exactly once. So the experiment needs to evaluate how well a method can match planes across different views.

Metrics: To measure how well these methods associate planes from two images, the experiment uses the performance metric IPAA (image pair association accuracy) by Cai et al. [67] that evaluates the image-pair level matching results. Written as IPAA-X, the value represents the fraction of image pairs with not less than X% of the planes associated correctly.

Baselines: The experiment compares the proposed method with modules from other systems as baselines. These results are measured during evaluation based on ground-truth bounding boxes to eliminate the interference of plane detection. Appearance Only runs the optimization only with the appearance embedding feature ${\mathbf{e}}_{1,j}$ predicted for each plane, which matches planes based on learning-based embeddings. Sparse Planes [17] runs its optimization without the relative camera pose as input.

Quantitative Results: Figure 17 shows that the proposed method outperforms baselines and other systems on IPAA-X metrics. It improves IPAA-80 by nearly 10% compared to Sparse Planes [17]. The Appearance Only baseline method only considers the texture of planes when matching; it can hardly reach 100% accuracy because it is very common for different objects to have similar textures in real-world scenarios. Sparse Planes occasionally reaches 100% accuracy but soon falls behind when a small portion of mismatches can be tolerated. The proposed method increases the successful matching rate for all accuracy levels; these improvements should come from more planes being matched due to the proposed method being able to leverage the relative camera pose. Detailed statistics are presented in

Table 3. Plane matching using ground-truth bounding boxes. IPAA-X [67] represents the fraction of image pairs with not less than X% of the planes associated successfully.

	IPAA-100	IPAA-90	IPAA-80
Appearance Only	6.8	23.5	55.7
Sparse Planes	16.2	28.1	55.3
Proposed	16.3	32.4	65.7

.

Qualitative Results: Comparison results are shown in Figure 18. Appearance Only does not take advantage of geometric information and struggles to distinguish between planes with similar textures. Without the help of the relative camera pose, Sparse Planes [17] sometimes fails to match plane segments with smaller surface areas. The proposed method distinguishes between planes with similar textures and matches plane segments of any size.

5. Conclusions

This paper proposed a novel method for 3D planar reconstruction from a pair of images using the relative camera pose. Compared to existing methods, the proposed method can take advantage of the relative camera pose to optimize full-scene reconstruction results further and to improve reconstruction quality when the camera angle between input images is less common. The method also improved plane matching accuracy with the help of the relative camera pose and increased the IPAA-80 by nearly 10% over the existing system.

The proposed method can provide planar surface reconstruction for 3D sensing or scene understanding for sparse view scenarios, with fewer sensors and input data required. To improve the scene reconstruction quality, future work can embrace a more robust plane prediction backbone to provide more accurate plane segmentation masks and parameters for the plane matching and merging modules. To expand the application’s scope, future work can focus on generalization and can evaluate the method on newly collected real-world datasets or more realistically rendered datasets. Last but not least, future work may also extend this architecture to a multi-view basis to reconstruct larger scenes in a progressive way.