Hash Indexing-Based Image Matching for 3D Reconstruction

Featured Application

Our method can be used for 3D reconstruction, metaverse, virtual reality and augmented reality.

Abstract

Image matching is a basic task in three-dimensional reconstruction, which, in recent years, has attracted extensive attention in academic and industrial circles. However, when dealing with large-scale image datasets, these methods have low accuracy and slow speeds. To improve the effectiveness of modern image matching methods, this paper proposes an image matching method for 3D reconstruction. The proposed method can obtain high matching accuracy through hash index in a very short amount of time. The core of hash matching includes two parts: creating the hash table and hash index. The former is used to encode local feature descriptors into hash codes, and the latter is used to search candidates for query feature points. In addition, the proposed method is extremely robust to image scaling and transformation by using various verifications. A comprehensive experiment was carried out using several challenging datasets to evaluate the performance of hash matching. Experimental results show that the HashMatch presents excellent results compared to the state-of-the-art methods in both computational efficiency and matching accuracy.

1. Introduction

In recent years, image matching has attracted much attention from both computer vision and artificial intelligence communities and has been used in various practical applications, such as 3D reconstruction [1], augmented reality, virtual reality [2,3], visual localization, visual tracking, object detection, image classification [4,5], and medical image analysis. In the case of 3D reconstruction [6], the components of image-based 3D reconstruction mainly consist of image matching, camera pose estimation, triangulation, and bundle adjustment. Thus, image matching is one of the most important modules in image-based 3D reconstruction [7]. The goal of image matching is to generate feature correspondences across multiple views. Once feature correspondences have been generated, the structure from motion (SFM) system can be used to recover sparse point clouds and camera parameters, then the multi-view stereo and Poisson surface reconstruction can be utilized to generate dense point clouds and texture models, respectively. As a result, we can conclude that the quality of 3D reconstruction heavily depends on image matching methods.

The standard pipeline of image matching consists of feature point detection, feature descriptor computing, feature matching, and outlier removal [8]. Each component of image matching has many potential choices. In the case of feature point detection, there are scale-invariant feature transform (SIFT) [9], speeded-up robust features (SURF) [10], KAZE features [11], oriented fast and rotated brief (ORB) [12], and learned invariant feature transform (LIFT) [13]. These features can not only locate feature points but also generate feature descriptors. Moreover, there are also some deep learning-based methods, such as boosted efficient local image descriptor (BELID) [14], repeatable and reliable detector and descriptor (R2D2) [15], joint description and detection of local features (D2Net) [16], and SuperGlue [17]. According to reports in corresponding papers, every method has a state-of-the-art performance. For outlier removal, the ratio test [9] is the simplest but most efficient method, and it uses a constant threshold to reject incorrect feature correspondences and is widely used in various computer vision tasks. Ma et al. [18] proposed an automatic approach to remove outliers by vector field learning. This method has an excellent performance even when dealing with large-scale feature collections with heavy outliers.

For feature matching, there are also many methods. For example, brute force matching (BFM) [9] is one of the most famous methods. For a given query point, BFM chose a maximal confident candidate from reference points as the corresponding point. Usually, the feature correspondences generated by BFM have some outliers, often caused by the feature descriptor’s ambiguity. Thus, BFM combined with the ratio test is a good solution for image matching. Wang et al. [19] proposed the use of a graph-matching approach to generate feature correspondences in the presence of outliers. The core idea is to reject incorrect matches by using the zero-assignment constraint. Jiang et al. [20] proposed the use of spatial clustering to generate a robust image matching result. The key idea of this method is to progressively cluster the putative matches into several motion-consistent clusters, including an outlier cluster. Zhou et al. [21] proposed progressive large-scale-invariant image matching in scale space for 3D reconstruction, resulting in a desirable performance in matching precision. Han et al. [22] proposed a deep learning-based approach to image matching as they designed a unified solution for learning feature representations and learning feature comparison functions. Although image matching has made significant progress, the topic still needs more attention, especially in image-based 3D reconstruction, and there is still much room for improvement in the accuracy and time efficiency of image matching methods.

After a deep investigation into the topic of image matching, we found that hash indexing is an efficient approach to feature descriptor matching. As a result, in this paper, we propose the design of a HashMatch based on hash indexing for image matching in 3D reconstruction. The HashMatch consists of two modules. First, constructing a big hash table for located feature points via Haar wavelet feature transform [23]. Second, indexing a potential candidate by using a short hash code for the query point. The HashMatch can significantly reduce computational costs on image matching when dealing with big image datasets. According to our experiments, the HashMatch has not only proved to be extremely fast but also has the desired matching precision and is very suitable for the processing of large-scale 3D reconstruction. In summary, the main contributions of this work are summarized as follows:

• We propose the use of hash indexing to replace the original brute force matching for image matching and present a HashMatch for 3D reconstruction.
• We propose the use of the Haar wavelet feature transformation to construct a big and robust hash table for the located feature points for feature descriptor indexing.
• We have designed a parallel architecture for hash matching to speed up the image matching process and save computing time. Additionally, a systematic experiment conducted on several benchmarking datasets is provided to assess the HashMatch against the state-of-the-art methods.

The rest of this paper is organized as follows: After a brief introduction in Section 1, related work about image matching is presented in Section 2. In Section 3, we present the HashMatch in detail. In Section 4, a systematic experiment was conducted on several benchmarking datasets to prove the effectiveness of the HashMatch. Finally, the conclusion and remarking comments are presented in Section 5.

2. Related Works

In this section, we will briefly review the most relative work to our research topic, including image matching, 3D reconstruction with SFM, MVS, and surface reconstruction.

2.1. Image Matching

Image matching has obtained great progress in the past decades due to many potential applications, so various feature-matching methods have been proposed by researchers [24,25]. In a nutshell, existing methods can be roughly divided into two categories according to the types of input data (video, and images). If the input data are in video form, the KLT and variants are usually adopted to locate feature correspondences by computing the displacements between two consecutive frames [26]. The Lucas and Kanade tracker (KLT) [27] is generally considered to be the best choice for video feature tracking, especially in robotic navigation, trajectory extraction for moving objects, and visual object tracking. Zach et al. [28] presented an adaptive KLT that was implemented in parallel programming for real-time applications. Hwangbo et al. [29] proposed the use of the inertial measurement unit (IMU) to improve the stability of KLT for a moving camera. Poling et al. [30] presented a novel framework for jointly tracking a collection of feature points, which can share information between the different feature points in the scene. This method achieves stable feature tracking via subspace constraints. Zhao et al. [31] formulated the feature tracking problem as a spatio-temporal statistical learning framework, so they proposed a robust keypoint tracking method via metric learning-driven multi-task structured output optimization. Generally, KLT and its variants can produce promising results for video sequences.

Regarding the second category, there have also been many approaches published in recent years. For example, Zhang et al. [31] proposed an efficient feature-tracking method to process a nonconsecutive image sequence for large-scale SFM, in which the parallel SIFT feature and second-pass matching strategy were adopted to handle feature-tracking drift. Lin et al. [32] developed a novel method called RepMatch for reconstructing the city model, in which the Epipolar guided approach is adopted to accommodate both wide baselines and repeated structures. Additionally, the RepMatch also utilized random sample consistency (RANSAC) to handle different true or false feature correspondences. Lukos et al. [33] proposed wide-baseline image matching via projective view synthesis for two-view image matching. This method can aptly handle the limitations caused by the approximate character of affine transformations via projective transformations, and can also use the essential matrix between images to reject incorrect feature correspondences. Recently, Lowry et al. [34] proposed to make use of local geometric priors to boost the spatial verification for filtering outliers. This method also utilized the scale and orientation information from feature neighborhoods to identify which feature points are likely to be correct feature correspondences and has proved to be excellent in practice.

Recently, the deep learning technique has provided excellent performances in various computer vision tasks, such as object detection and recognition, visual tracking and localization, and depth estimation. Thus, some researchers have tried to use deep learning techniques to handle image matching, including feature point detection, descriptor computing, and feature matching. For example, the first work of deep learning-based image matching was MatchNet, proposed by Han et al. [22], in which an end-to-end neural network architecture was designed to generate feature correspondences for image pairs. Zhao et al. [35] presented a hierarchical neural network named NM-Net to generate high-precision feature correspondences via mining reliable neighbors, in which NM-Net takes the generated graph as an input, then boosts the robustness to the order of feature correspondences. Yi et al. [36] designed a deep neural network architecture for the purpose of learning to locate good feature correspondences for wide-baseline images. They train the network in an end-to-end solution to label the feature correspondences as inliers or outliers. Sarlin et al. [17] proposed a neural network named SuperGlue to match two sets of local feature points by jointly finding feature correspondences and rejecting incorrect matches. Additionally, they also presented a flexible context aggregation mechanism by making use of visual attention to enable SuperGlue to reason the underlying geometry model. As a result, SuperGlue is an excellent solution based on deep learning for image matching. Ma et al. [37] presented a systematic survey to analyze the research state of image matching. More information about image matching can be found in Ma’s work.

2.2. 3D Reconstruction

Reconstructing geometric models from image sets is a basic task in the fields of computer vision and computer graphics, which has led to the emergence of various 3D reconstruction methods. The classic pipeline of image-based 3D reconstruction consists of sparse, dense, and surface reconstruction. Each component has different solutions or methods. The goal of sparse reconstruction is to recover sparse point clouds and camera parameters from the generated feature correspondences via SFMs. For instance, Snavely et al. [38] developed a system named Bundler to recover sparse geometry from Internet image collections. Bundler possesses an excellent implementation and has the robustness to scene scales. Wu et al. [39] present a novel SFM system that reaches linear-time reconstruction via incremental way with the parallel SIFT and multi-core bundle adjustment. In addition to fast speeds, Wu’s system can generate compact point clouds by re-triangulation for drifted features. Chris et al. [40] presented a large-scale SFM for outdoor scene reconstruction, in which they make use of the distributed camera model to describe image observations in terms of light rays rather than image pixels. Zhu et al. [41] developed a very large-scale global SFM to replace incremental SFMs, in which distributed motion averaging is utilized to estimate the camera pose in a parallel manner. Johannes et al. [42] revised the development progress of SFMs, then designed an excellent incremental SFM via scene graph augmentation. According to the reports in [43,44], modern SFMs can recover a world-scale model from Internet image collections in a few days.

For dense reconstruction, the patch-based multi-view stereo (PMVS) [45] is an excellent solution, which takes local feature points and edges as seeds to densify the sparse point clouds to generate dense point clouds via point expansion. Goesele et al. [46] proposed a depth-map-fusion-based MVS for generating dense point clouds. The method first computes individual depth maps using a stereo matching approach, then the depth maps are merged into a single dense model using a direct volumetric fusion. Silvano et al. [47] developed a massively parallel method for high-quality multi-view stereo matching, with the core idea being based on the PathMatch, which uses a robust photo-consistency measure to check depth errors. Kuhn et al. [48] proposed a scalable and efficient MVS method to reconstruct an accurate dense geometry model from hundreds of high-resolution images, in which all depth maps are estimated by semi-global matching (SGM) methods and a TV previously utilized to improve the quality of MVS. Inspired by the success of deep learning in pattern recognition, some researchers have tried to design neural networks for MVS reconstruction with end-to-end approaches. For example, Yao et al. [49] proposed the use of a deep learning-based approach to inference depth maps for generating dense point clouds. This resulted in the construction of a novel neural network architecture named MVSNet. Later, they modified MVSNet with recurrent neural networks (RNN), resulting in RMVSNet [50].

Once dense point clouds have been generated, surface reconstruction methods can be utilized to generate texture models. Among the existing methods, the most famous method is Poisson surface reconstruction [51], which formulates the problem of reconstructing the textured model as the Poisson equation solver. Zhou et al. [52] proposed the use of data-parallel Octrees to accelerate Poisson surface reconstruction when dealing with large-scale scenes, subsequently resulting in parallel Poisson surface reconstruction. Waechter et al. [53] presented a novel approach to generating high-quality textures for large-scale 3D reconstructions. This method can handle the complexity and scale of reconstructed models. Ummenhofer et al. [54] developed a variational method for surface reconstruction from the MVS data, which can handle a billion points in a short time. Recently, Tom et al. [55] presented a BigSUR for large-scale structured urban reconstruction, in which a binary integer program is proposed for global balances sources of error to generate semantically parsed mass models with associated façade elements. With the rapid development of deep learning technology, some learning-based approaches have been proposed for reconstructing texture models from image collections via end-to-end ways, such as SurfaceNet [56]. We believe that more efficient methods based on deep learning will be proposed in the future.

3. The Proposed Method

To improve the quality of modern 3D reconstruction, in this paper, we present an efficient image matching method based on hashing indexing, named HashMatch. The pipeline of the HashMatch is depicted in Figure 1, where we first use the SURF feature to find local feature points and compute feature descriptors for given image pairs. Second, Haar wavelet coding is adopted to construct a compact hash table for the located feature points and descriptors. Thirdly, the hash index replaces the traditional KNN to select the corresponding feature point from the reference feature set, and then the wrong feature correspondences are caused by repeating the query process. Note that the ratio test is also integrated into the HashMatch for rejecting outliers. The procedures of the HashMatch are presented in Algorithm 1. More details can be found in the subsequent sections.

Algorithm 1 HashMatch.

Input: Image collection $I=\left\{I_n|n\in \left[1,N\right]\right\}$ $\mathrm{Output}: \mathrm{Feature} \mathrm{correspondences} M_{i,j}=\left\{f\left(p,q\right)|p,q\in \left[1,M\right]\right\}$

Step 1: Use the SURF feature to detect feature points for each image in image collection $I$. Step 2: Use the SURF feature to compute feature descriptors for each located feature point. Step 3: Create a hash table for all feature descriptors:    for idx = 1 to $N^{\prime }$:    (1) Compute Haar wavelet coefficients for each descriptor via Equation (8);    (2) Calculate the values of $e_{k1}$,$e_{k2}$, $e_{k3}$, and $e_{k4}$ via Equations (9)–(12);    (3) Assign values to the members of hash table htable via Equations (13)–(16); Step 4: Indexing hash table for given numbers $i$ and $j$:    (1) Compute the values of $i{d}_0$, $i{d}_1$, and $i{d}_2$ according to Equation (18);    (2) Calculate the indexing table for locating feature descriptors via Equation (19); Step 5: Measure the difference between feature descriptor $d_p$ and $d_q$ via L2-distance; Step 6: Repeat Step 4~Step 5. This should result in a set of feature correspondences for each image pairs.

3.1. Feature Detection

Many feature point detectors can be candidates for image matching in 3D reconstruction, such as SIFT [9], ORB [12], KAZE [11], SuperPoint [57], and SuperGlue [17]. In this paper, the SURF [10] feature is adopted to find feature points and compute feature descriptors due to its fast speed and comparable matching precision to SIFT. Figure 2 presents the pipeline of the SURF detector for locating feature points from the given image. Specifically, for the input image $I$, the box filter is utilized to construct multi-scale spaces via different kernel sizes, and the integral image operation is also adopted to accelerate convolutional operation.

$$I_{\propto }\left(x,y\right)={\displaystyle \sum }_{\begin{array}{c}0\le x^{\prime }<kernel.cols\\ 0\le y^{\prime }<kernel.rows\end{array}}kernel\left(x^{\prime },y^{\prime }\right)\ast I\left(x+x^{\prime }-anchor.x,y+y^{\prime }-anchor.y\right)$$

(1)

The position of the feature point can be computed by the Hessian matrix in multi-scale spaces. The Hessian matrix $H\left(x,y,\sigma \right)$ in $\left(x,y\right)$ at scale, $\sigma$ is defined as

$$H\left(x,y,\sigma \right)=\left[\begin{array}{cc}{L}_{xx}\left(x,y,\sigma \right)& L_{xy}\left(x,y,\sigma \right)\\ L_{xy}\left(x,y,\sigma \right)& L_{yy}\left(x,y,\sigma \right)\end{array}\right]$$

(2)

where $L_{xx}\left(x,y,\sigma \right)$ is the convolution of the Gaussian second-order derivative $G\left(x,y,\sigma \right)$ with image $I$ in point $\left(x,y\right)$.

$$L_{xx}\left(x,y,\sigma \right)=\frac{{\partial }^{2}g\left(x,y,\sigma \right)}{x^2}\otimes I\left(x,y,\sigma \right)$$

(3)

Notice that $L_{xy}\left(x,y,\sigma \right)$ and $L_{yy}\left(x,y,\sigma \right)$ can be calculated similarly.

To this end, we can judge whether $\left(x,y\right)$ is a feature point according to the approximate determinant Det($H^{\prime }\left(x,y,\sigma \right)$) of the Hessian matrix $H\left(x,y,\sigma \right)$.

$$\mathrm{Det}(H^{\prime }\left(x,y,\sigma \right))=L_{xx}\left(x,y,\sigma \right)L_{yy}\left(x,y,\sigma \right)-{\left(0.9L_{xy}\left(x,y,\sigma \right)\right)}^2$$

(4)

If $\mathrm{Det}\left(H^{\prime }\left(x,y,\sigma \right)\right)$ is greater than all pixel values in the neighborhood of $\left(x,y\right)$, then $\left(x,y\right)$ is a feature point, otherwise, it is not. As depicted in Figure 2, the circles containing yellow are the located feature points.

3.2. Descriptor Extraction

The descriptor is a representation of the local feature point, some novel descriptors have been proposed in recent years. For example, Lowe’s SIFT and its variants, AKAZE feature [11], local difference binary (LDB) [58], locally uniform comparison image descriptor (LUCID) [59], learned arrangements of three patch codes (LATCH) [60], learned invariant feature transform (LIFT) [13], and SURF [10]. Binary descriptors possess a fast computational efficiency but have low matching precision. Floating-point descriptors have high matching precision but slow speed. There is an exception, namely SURF, which is a floating-point descriptor extractor, but it has comparable computational efficiency to binary descriptor extractors such as ORB and LDB. As a result, in this paper, we also use SURF to compute feature descriptors for located feature points due to it providing the best balance between distinctiveness and computational cost.

Figure 3 presents the pipeline for extracting feature descriptors for the located feature points via the SURF feature descriptor extractor. For the given feature point $\mathrm{x}=\left(x,y\right)$ and corresponding image $I$, we first compute a dominant orientation $\theta$ for feature point $\mathrm{x}$ by computing Haar wavelet responses in the neighborhood centered at $\mathrm{x}$ with radius $6s$, where $s=1.2h/9$ and $h$ is the size of the current filter kernel. With the dominant orientation $\theta$, we crop the patch $p$ with the size of $20s\times 20s$ centered at $\mathrm{x}$ from image $I$, then split the patch $p$ into 16 sub-regions along with the dominant orientation $\theta$, as depicted in Figure 3c, where each sub-region has four-types of Haar wavelet responses, namely

$$Res{p}_i=\left\{{{\displaystyle \sum }}_{i}dx,{{\displaystyle \sum }}_i\left|dx\right|,{{\displaystyle \sum }}_{i}dy,{{\displaystyle \sum }}_i\left|dy\right|\right\}$$

(5)

where $i$ denotes the index of sub-regions.

To this end, we can obtain a floating-point feature vector with 64 elements.

$$D_x=\left\{Res{p}_i|i=1,2,\cdots ,16\right\}$$

(6)

Furthermore, we can normalize $D_x$ by the Frobenius normal to boost its robustness to light and scale changes, and repeated texture.

3.3. Hash Matching

The goal of feature descriptor matching is to find feature correspondences for given independent feature collections. In the past decade, many approaches to feature descriptor matching have been proposed by researchers from academia. The simplest method is a k-nearest neighbor (KNN) matching with a ratio test. However, KNN is time-consuming, especially in large-scale feature collections. Johannes et al. [42] formulate feature descriptor matching as image searching for large-scale 3D reconstruction to reduce the computational cost of image matching, but matching precision may decrease in practice. Although there are some GPU-accelerated methods for image matching, they can also be improved in terms of both computational burden and matching precision. To improve the quality of image matching for boosting modern 3D reconstruction, in this paper, we propose a simple but efficient approach to feature descriptors matching via hashing indexing; thus, the proposed method is named hash indexing.

The pipeline of hash indexing is depicted in Figure 4, where feature descriptors are computed by the SURF feature [10]. For a given set of feature descriptors, the following steps should be conducted to generate feature correspondences: (1) create a hash table for all feature descriptors that included the initial hash table, feature transformation, and updated feature bins; (2) perform hash matching, specifically searching feature correspondences using the hash value of feature descriptors. In the implementation, we can make use of the Haar wavelet to create an approximate hash table for the acceleration of feature descriptors that match further. According to the report in Section 4, the proposed method outperforms the state-of-the-art methods in both efficiency and accuracy. It is salient to note that hash matching belongs to similar matching rather than brute-force matching; thus, it has a faster speed and desired matching precision.

3.3.1. Creating a Hash Table

Descriptor matching is still a challenging task in large-scale 3D reconstruction. Although some novel approaches have been proposed for accelerating descriptor matching in image matching, these methods need further improvements regarding how they deal with high-resolution images. To reduce the computational costs of descriptor matching, in this section, we propose the creation of a big hash table to accelerate image matching via hash indexing, then boost the computational efficiency of image-based 3D reconstruction. Specifically, let $htable$ be a hash table for the given descriptor collections $d=\left\{d_1,\cdots ,d_N\right\}$, and the $htable$ is defined as (in C-Plus-Plus language):

$$htable=\left\{int N;float sum\left[3\right];float avg\left[3\right];float squsum\left[3\right];float std\left[3\right]\right\}$$

(7)

where $N$ is the number of feature points, $sum\left[3\right]$ is used to store the sum of Haar wavelet coefficients of all feature descriptors and $squsum\left[3\right]$ is utilized to store the squared sum of Haar wavelet coefficients of all feature descriptors. Let $c_{k1}$, $c_{k2}$, and $c_{k3}$ represent the Haar wavelet coefficients for the descriptor $d_k\in d$, which can then be calculated using the following formulas,

$$\{\begin{array}{c}{c}_{k1}=e_{k1}+e_{k2}-e_{k3}-e_{k4} \\ c_{k2}=-e_{k1}+e_{k2}-e_3+e_{k4}\\ c_{k3}=e_{k1}-e_{k2}-e_{k3}+e_{k4} \end{array}$$

(8)

Additionally, $e_{k1}$,$e_{k2}$, $e_{k3}$, and $e_{k4}$ are the sum of descriptor elements and are defined as

$$e_{k1}={\displaystyle \sum }_{i=0}^{i=3}{\displaystyle \sum }_{j=0}^{j=3}{d}_k\left[i\ast 8+j\right]$$

(9)

$$e_{k2}={\displaystyle \sum }_{i=0}^{i=3}{\displaystyle \sum }_{j=4}^{j=7}{d}_k\left[i\ast 8+j\right]$$

(10)

$$e_{k3}={\displaystyle \sum }_{i=4}^{i=7}{\displaystyle \sum }_{j=0}^{j=3}{d}_k\left[i\ast 8+j\right]$$

(11)

$$e_{k4}={\displaystyle \sum }_{i=4}^{i=7}{\displaystyle \sum }_{j=4}^{j=7}{d}_k\left[i\ast 8+j\right]$$

(12)

Upon finding the values of $c_{k1}$, $c_{k2}$, and $c_{k3}$, the members of htable can be computed as

$$sum\left[index\right]={\displaystyle \sum }_{k=0}^{k=N}{c}_{k1},index=0,1,2$$

(13)

$$squsum\left[index\right]={\displaystyle \sum }_{k=0}^{k=N}{c}_{k1}\ast c_{k1},index=0,1,2$$

(14)

$$std\left[index\right]=\sqrt{\frac{squsum\left[index\right]}{N}-sum\left[index\right]\ast sum\left[index\right]},index=0,1,2$$

(15)

$$avg\left[index\right]=\frac{sum\left[index\right]}{N},index=0,1,2$$

(16)

To this end, we can create a big hash table based on Haar wavelet transformation for the input feature descriptor collections for fast indexing in the image matching stage.

3.3.2. Hash Indexing

The final stage of hash matching is to index hash table $htable$ for the given feature descriptor $d_k\in d$. Brute force searching is the simplest method for indexing the hash table, but it may demand high computational costs. To mitigate this, we propose the construction of an index table $indtable$ for the purpose of accelerating image matching. The $indtable$ is a multi-dimensional array, which can be defined as (in C Plus Plus language) $indtable:=std::vector\langle std::vector\langle std::vector\langle int\rangle \rangle \rangle ()$. Specifically, let $inv\_st{d}_{index}$=$1/st{d}_{index},index=0, 1, 2$, and $i{d}_0$, $i{d}_1$, and $i{d}_2$ represent indexing numbers. $t_0$, $t_1$, and $t_2$ denote temporal variables, namely

$$\{\begin{array}{c}{t}_0=indinv\_st{d}_0\ast \left(c_{k0}-avg\left[0\right]\right) \\ t_1=indinv\_st{d}_1\ast \left(c_{k1}-avg\left[1\right]\right) \\ t_2=indinv\_st{d}_2\ast \left(c_{k2}-avg\left[2\right]\right) \end{array}$$

(17)

The values of $i{d}_0$, $i{d}_1$, and $i{d}_2$ can be calculated by the following formulas.

$$\{\begin{array}{c}i{d}_0=\min \left(9,\max \left(0,floor\left(1.5\ast \left(t_0+3.333\right)\right)\right)\right) \\ i{d}_1=\min \left(9,\max \left(0,floor\left(1.5\ast \left(t_1+3.333\right)\right)\right)\right) \\ i{d}_2=\min \left(9,\max \left(0,floor\left(1.5\ast \left(t_2+3.333\right)\right)\right)\right) \end{array}$$

(18)

where $\min \left(\mathrm{x},\mathrm{y}\right)$ and $\max \left(\mathrm{x},\mathrm{y}\right)$ represents the minimal and maximal values between $\mathrm{x}$ and $\mathrm{y}$, respectively. To obtain an integer that is less than $x$, $floor\left(x\right)$ is used. Let $bins=std::vector\langle std::vector\langle int\rangle \rangle ()$, and $inde{x}_t=100\ast i{d}_0+10\ast i{d}_1+i{d}_2$. Thus, $bins\left[inde{x}_t\right]=k$, where $k$ represents the indexing number of the $k$th feature descriptor in the image $I_n, n\in \left[0,N-1\right]$. As a result, the value of the indexing table can be assigned using the following formula.

$$indtable\left[n\right]=bin{s}_n, n\in \left[0,N-1\right]$$

(19)

Once we have the indexing table, we can quickly obtain the descriptor collections for the image $I_n$ and image $I_{n+1}$ via hash indexing. The remaining operations of feature descriptor matching are similar to [23]. More detailed information which demonstrates the performance of the HashMatch can be found in Section 4, where a comprehensive experiment conducted on several challenging datasets is provided for readers.

4. Experimental Results

The HashMatch was implemented in Visual C++ 2017 with OpenCV SDK 4.1, and Nvidia CUDA SDK 10.0 version was also used to accelerate the process of image matching. Furthermore, we evaluated the HashMatch on the benchmarking dataset and the open-source dataset and also made a comprehensive comparison to KNN + RatioTest [9], ENFT [61], MODS [62], RepMatch [32], CODE [63], D2D [64], and R2D2 [15]. It should be noted that, during the experiment, each algorithm used the original parameters.

4.1. Evaluation on the EdgeFoci Dataset

The EdgeFoci dataset [65] is a benchmark for evaluating the performance of local feature detectors and descriptors. It consists of nine sequences and each sequence has a variant number of images. The samples of the EdgeFoci dataset are provided in Figure 5, they are Boat, Graffiti, Light, Dame, Obama, Painted Ladies, Rushmore, and Yosemite, respectively. The partially matching results are provided in Figure 6 where the BFM has the most incorrect matches. The RatioTest can reject some outliers from the matching collection produced by the BFM. The ENFT has more robustness than that of the BFM and KNN + RatioTest due to the use of a two-pass matching strategy. The HashMatch can generate the most accurate feature correspondences via Haar wavelet encoding. In the case of Notre Dame, it has symmetric structures and repeated features. The BFM produced the highest number of feature correspondences for Notre Dame, but there are too many incorrect matches in the collection of generated feature correspondences. The KNN + RatioTest also has outliers because its performance suffers from the constant threshold. If a big threshold is used by the KNN, it may produce more feature correspondences than that of the KNN with a small threshold but have more incorrect matches. Otherwise, The KNN with a small threshold can reject more incorrect feature correspondence; however, it can also reject some correct ones. Although the ENFT can filter many outliers from the collection of the initial feature correspondences by the two-pass matching strategy, a few incorrect matches also exist. However, the HashMatch has the best performance among the compared methods, providing the highest number of correct feature correspondences.

In addition to the qualitative analysis, we also provided the statistical results (number of correct feature correspondences) for the compared methods. The results are listed in

Table 1. Statistical results (correct matches) of the EdgeFoci dataset.

Method	Sequence
	Boat	Graffiti	Light	Notre Dame	Obama	Painted Ladies	Rushmore	Yosemite
KNN + RatioTest	95	70	501	70	21	39	17	23
ENFT [61]	141	96	698	89	25	80	23	28
MODS [62]	101	83	564	142	42	47	18	32
RepMatch [32]	117	120	721	137	60	69	35	41
CODE [63]	98	109	706	110	58	53	29	47
D2D + KNN [64]	167	203	792	89	81	90	42	59
R2D2 + KNN [15]	130	178	807	120	90	83	53	62
HashMatch (Ours)	558	378	1263	307	328	407	286	613

, where the higher the number of correct feature correspondences, the better the performance of the corresponding method. As the stated before, the KNN + RatioTest has the worst performance and the least amount of feature correspondences. The HashMatch has the highest number of feature correspondences (only inliers) on the whole dataset, demonstrating excellent performance. Finally, for each sequence, we recorded the computational time taken by the HashMatch. These times are depicted in Figure 7. The HashMatch can match ten thousand local features in a small amount of time. Moreover, we also provide the computational time for each method in Figure 8, where it can be seen that our proposed HashMatch has the fastest speed. To this end, both qualitative and quantitative results have proven the excellent performance of the HashMatch.

4.2. Evaluating HashMatch with Different Features

To further verify the effectiveness of the matching approach of the HashMatch, some different local features including BRISK, ORB, AKAZE, KAZE, SIFT, SURF & VGG, SIFT & VGG, have been integrated into the HashMatch, and we evaluated them using the Samper sequence of Strach’s dataset (http://cvlabwww.epfl.ch/data/multiview/denseMVS.html, accessed on 10 April 2019). The statistical results are listed in

Table 2. Statistical results (correct matches) of the Semper sequence of Strach’s dataset, where HM represents HashMatch, “SURF + VGG” represents SURF detector and VGG descriptor, and “SIFT + VGG” represents SIFT detector and VGG descriptor.

Quantitative Analysis	Method
	BRSK + HM	ORB + HM	AKAZE + HM	KAZE + HM	SIFT + HM	SURF + HM	SURF + VGG + HM	SIFT + VGG + HM
Features	19,808	20,000	11,896	11,434	38,991	45,944	45,944	38,991
Matches	3037	3857	4571	4699	6437	8539	7945	4808
Ratio of Inliers	79.34%	81.14%	88.56%	89.67	91.56	97.72	89.76	84.62

, where the “Features” denote the number of input local features, and the “Matches” represent the number of generated feature correspondences. As shown in Table 2, “SURF + HM” has the highest number of feature correspondences; thus, the effectiveness of the HashMatch is proved once again. In addition to statistical results, we also provide some visualized results in Figure 9, in which all combiners can produce accurate feature correspondence. This is an excellent property of various computer vision tasks, especially in 3D reconstruction based on images.

4.3. Evaluation on the Open-Source Dataset

To verify the effectiveness of the HashMatch in 3D reconstruction, we integrated it into the Bundler and evaluated it by using the open-source dataset (https://github.com/rperrot/ReconstructionDataSet, accessed on 12 April 2021). It should be noted that this dataset contains nine sequences. In our experiment, the “AvignonHotelDesMonnaies” (in the following section, we call it a “Hotel” for convenience) was adopted because it is the most challenging sequence. Figure 10 presents the samples of the Hotel sequence, there are many repeated features on the surfaces of the sampling images, and this raises a big challenge for the image matching method. The matching results of the compared methods are depicted in Figure 11, where it can be seen that the BFM has the most incorrect feature correspondences, meaning that BFM had the lowest matching precision among these image matching methods. Although the RatioTest can remove some outliers, there are also many incorrect feature correspondences in the collection of matches. The ENFT obtained a better result than that of the KNN + RatioTest because of its use of the two-pass matching strategy. The HashMatch yielded the best results among these methods as there were no outliers in the collection of feature correspondences.

Three ways are adopted to evaluate the performance of the HashMatch in 3D reconstruction, namely sparse reconstruction by Bundler, dense reconstruction by the PMVS, and surface reconstruction via the Poisson surface reconstruction. The sparse point clouds, dense point clouds, and textured models are provided in Figure 12, Figure 13 and Figure 14, respectively. In each stage, the reconstructed model had a high geometry consistency with the real scene. Moreover, the HashMatch to one minute to generate feature correspondences. To this end, all of our results (Figure 11, Figure 12, Figure 13 and Figure 14) prove the effectiveness of the HashMatch in 3D reconstruction.

4.4. Evaluation on Strecha’s Dataset

To further assess the effectiveness of the HashMatch in 3D reconstruction, we integrated it into COLMAP and evaluated its effectiveness by using Strecha’s dataset [66]. It should be noted that this dataset contains six sequences. In our experiment, the “Herz-Jesu-P25” is used because it is the most challenging sequence. Figure 15 presents the samples of Herz-Jesu-P25, and it can be observed that there are many repeated features on the surfaces of the sampling images, and this raises a big challenge for the image matching methods. The matching results of the HashMatch are depicted in Figure 16. Only correct feature correspondences are visible, indicating that the HashMatch had the highest matching precision compared to state-of-the-art methods. In summary, the proposed HashMatch has the best performance matching precision.

Three ways are used to evaluate the performance of the HashMatch in 3D reconstruction, namely sparse reconstruction by COLMAP, dense reconstruction by the PMVS, and surface reconstruction via the Poisson surface reconstruction. The sparse point clouds, dense point clouds, and textured models are provided in Figure 17, Figure 18 and Figure 19, respectively, and in each stage, the reconstructed model has high geometry consistency with the real scene. Moreover, the HashMatch only took a few minutes (0.75 min) to generate feature correspondences. To this end, all of our results (Figure 16, Figure 17, Figure 18 and Figure 19) prove the effectiveness of the HashMatch in 3D reconstruction.

5. Conclusions

In this article, we propose an efficient image matching method for 3-D reconstruction. The proposed method was named HashMatch because it makes use of hash indexing to achieve large-scale feature matching in a short amount of time. We adopted the SURF feature to describe images, resulting in high-quality feature points and robust feature descriptors. We suggest using hash coding to generate a large hash table for the image set, so as to quickly index. Multiple verifications are utilized to reject incorrect feature correspondences. The results of systematic experiments show that the HashMatch is better than the state-of-the-art methods as it had the highest matching precision on all challenging datasets. In addition to this, the proposed method has good extensibility, which can work well with different feature detectors or feature descriptors. In short, our method is fast and accurate.

In the future, we also plan to design parallel architecture with the CUDA SDK for the HashMatch to provide more generic, fast, and accurate solutions for real-time image matching.