Adaptive Image Matching Using Discrimination of Deformable Objects

Abstract

We propose an efficient image-matching method for deformable-object image matching using discrimination of deformable objects and geometric similarity clustering between feature-matching pairs. A deformable transformation maintains a particular form in the whole image, despite local and irregular deformations. Therefore, the matching information is statistically analyzed to calculate the possibility of deformable transformations, and the images can be identified using the proposed method. In addition, a method for matching deformable object images is proposed, which clusters matching pairs with similar types of geometric deformations. Discrimination of deformable images showed about 90% accuracy, and the proposed deformable image-matching method showed an average 89% success rate and 91% accuracy with various transformations. Therefore, the proposed method robustly matches images, even with various kinds of deformation that can occur in them.

1. Introduction

Computer vision lets a machine or computer see and understand objects, just like human vision. The purpose of computer vision is to recognize object in an image and/or to understand the relationships between objects. While recent research has focused on recognition based on big data and deep learning (DL) [1], traditional computer vision methods are still widely used in some specific areas. While DL is not yet used in many applications due to the requirement for high computing power and big data, traditional research based on hand-craft techniques, like feature detection and feature matching, are actively used in various applications, such as machine vision, image stitching, object tracking, and augmented reality. Image matching is matching similar images or objects, even under geometric transformations, such as translation; optical, scale, and rotation transformations; affine transformations, which are complex transformations; and viewpoint change. The new current challenge is image matching of deformable objects [2]. In the real world, deformable objects encompass the majority of all objects. In particular, research related to fashion items, such as clothes, is conducted because of the rapid growth in Internet shopping [3]. However, since deformable-object matching has a different target than the existing image-matching methods, feature-detection and -matching methods are not identical. As such, much research has been conducted into achieving both objectives at the same time, but without substantial results. Since deformable models cannot be defined by a specific transformation model, there are numerous difficulties in such research. Early deformable matching methods were researched for augmented reality, remote exploration, and image registration for medical images, and it was not until recently that research on deformable-object matching appeared.

A good matching algorithm is characterized by robustness, independence, and a fast matching speed [4]. Robustness is recognizing that two images are identical if they have exactly the same objects in them. However, the algorithm must recognize identical objects even under transformation. Independence is recognizing the differences between two images containing objects that are different. Lastly, fast matching is the property of rapidly determining a match. Without fast matching, an algorithm cannot process many images, and hence, cannot be a good algorithm. The biggest disadvantage to previous deformable object-matching algorithms is slow matching. Therefore, in this paper, as a solution to the problems of the aforementioned methods (and by considering these characteristics), we propose an efficient algorithm that operates the same way for both rigid and deformable objects.

The rest of this paper is composed as follows. Section 2 introduces the existing image-matching methods. Section 3 introduces the proposed algorithm, followed by Section 4, where the experimental results from image sets with various deformable objects are examined and analyzed. Finally, Section 5 evaluates the proposed algorithm, and concludes the paper.

2. Related Works

In this section, we introduce the known feature-matching methods for computer vision. Image-matching methods can be largely classified into matching rigid objects and matching deformable objects. Rigid object-matching methods mostly consist of those that examine geometric relationships based on feature correspondence, and that show good performance under transformations like viewpoint change and affine transformation. However, matching performance degrades when deformable objects are the target. For deformable object-matching, various methods are used depending on the specific image set. In other words, there is no generalized procedure. Moreover, there is the common problem of generally slow execution time. We introduce three categories of common feature-point matching methods, which are classified in terms of methodology [5].

2.1. Neighborhood-Based Matching

Early researchers used neighbor-pixel information around the points to find feature correspondence. Neighborhood-based methods include threshold-based, nearest neighbor (NN), and nearest neighbor distance ratio (NNDR) [6]. In the threshold-based method, if the distance between the descriptors is below a predefined threshold value, the features are considered to be matched. The problem in this method is that a single point in the source image may have several matched points in the target image. In the NN approach, two points are matched if their descriptor is in the nearest neighborhood and the distance is below a specific threshold. Therefore, the problem with the threshold-based method is solved. However, this method may result in many false positives between images, known as outliers. This is overcome by random sample consensus (RANSAC), which was proposed by Fischler and Bolles [7]. RANSAC is a good outlier-removal method, and is widely used for object detection and feature matching.

2.2. Statistical-Based Matching

Logarithmic distance ratio (LDR), proposed by Tsai et al. [8], is a technique for fast image indexing, which calculates the similarities in feature properties, including scale, orientation and position, and recognizes similar images based on lower similarity distance. This technique is effectively used to rapidly search similar image candidates in large image datasets. Lepsøy et al. proposed the distance ratio statistic (DISTRAT) method [9], which adopts LDR for image matching. They found that the LDR of inlier matching pairs has a specific ratio when LDR is calculated using feature coordinates only. Therefore, the final matching decision is based on a statistical analysis whereby the LDR histogram is narrower with more inliers. This method has the advantage of performance equivalent to RANSAC while having a faster matching speed, and was eventually selected as the standard in Moving Picture Expert Group (MPEG)—Compact Descriptors for Visual Search (CDVS) [10].

2.3. Deformable Object-Based Matching

The aforementioned matching algorithms are optimized for rigid objects. However, most real-world objects are deformable. Previously proposed feature-based deformable object-matching methods include transformation model-based [11], mesh-based [12], feature clustering [13], and graph-matching [14] methods. Transformation model-based methods require high complexity, because they operate under a pixel-based model. Therefore, feature-based methods that are not pixel-based were proposed. Agglomerative correspondence clustering (ACC), proposed by Cho and Lee [15], is a method that calculates the correspondence between each matching pair, and clusters matching pairs with a similar feature correspondence. ACC defines feature correspondence as the difference between the points calculated using the homograph model of matching pairs and the point that was actually matched. Then, matching pairs with a small difference are considered to have a locally similar homography, and are hierarchically clustered. Unlike previous methods, ACC will cluster matching pairs with a similar homograph by calculating the geometric similarity between each matching pair, rather than classifying matching pairs into inliers and outliers. While ACC has the disadvantages of appreciably high complexity and a high false positive rate, it shows the best performance when only the true rate is considered. As such, Improved ACC was introduced as an enhanced version of ACC [16].

3. Proposed Algorithm

RANSAC shows robust performance against geometric transformation of rigid objects. However, it does not offer good performance when matching deformable objects. Meanwhile, deformable object-matching methods generally have high complexity, which presents difficulty in applications that require fast matching. The easiest solution to this issue is to first perform rigid object-matching and match the remaining non-matched points using deformable object-matching methods. However, this is an inefficient solution. Therefore, it is more effective to selectively adopt the matching method, as long as it can discriminate deformable objects. As a solution, we propose discrimination of deformable transformations based on statistical analysis of matching pairs, and the subsequent use of the corresponding matching method. For example, if there are no inliers at all from among numerous matching pairs, then these are likely to be non-matching pairs. Moreover, even if there are some inliers, they are unlikely to be a match if the inlier ratio is low. Since the inlier ratio is low for deformable objects, it is impossible to obtain good results. This paper proposes discrimination of possible deformable objects through statistical analysis of such matching information and supervised learning.

Figure 1 shows a diagrammatic representation of the proposed image-matching process. First of all, features are detected from the image, and candidate matching pairs are found. Final matching is examined through geometric verification. The algorithm is an adaptive matching method wherein the possibility of the candidate matching pair being a deformable object is examined through discrimination of the deformable object (using the proposed algorithm) within the candidate matching pairs that do not satisfy geometric verification, Deformable object-matching is only performed on matching pairs that are determined to be deformable objects. If the algorithm cannot discriminate the deformable objects well, it is an inefficient algorithm. Therefore, how well the algorithm discriminates deformable objects significantly affects the overall performance.

3.1. Feature Detection, and Making a Matched Pair

The proposed method requires feature coordinates and properties for deformable-object matching. Therefore, a scale-invariant feature transform (SIFT) detector is used, which is a point-based detector that includes the feature’s coordinates, scale, and dominant orientation [17]. Speeded up robust features (SURF), maximally stable extremal regions (MSER), and affine detectors, which have similar properties, can also be used. SIFT is a set of orientation histograms created on a 4 × 4-pixel neighborhood with eight bins each. The descriptor then becomes a vector of all the values of these histograms. Since there are 4 × 4 = 16 histograms, each with eight bins, the vector has 128 elements. The features extracted from each image are $F\left(i\right)=\left\{c_i,s_i, o_i,des{c}_i\right\} \left(i=0~N\right)$ where $c_i$ is a coordinate, $s_i$ is the scale, $o_i$ is the dominant orientation, and $des{c}_i$ denotes 128-dimensional descriptors. To form matching pairs, the features detected from each image are compared. The metric used for the comparison is Euclidean distance, which is given by Equation (1):

$$d\left(f_{R\left(i\right)},f_{Q\left(j\right)}\right)=\sqrt{{\displaystyle \sum }_{k=1}^{128}{(f_{R\left(i\right)}^k-f_{Q\left(j\right)}^k)}^2}$$

(1)

which obtains the Euclidean distance between the i-th feature vector of the reference image, $f_{R\left(i\right)}$, and the j-th feature vector of the query image, $f_{Q\left(j\right)}$. The simplest feature matching sets a threshold (maximum distance) and returns all matches from other images within this threshold. However, the problem with using a fixed threshold is that it is difficult to set; the useful range of thresholds can vary greatly as we move to different parts of the feature space [18]. Accordingly, we used NNDR for feature matching [17] as follows:

$$NNDR=\frac{d_1}{d_2}=\frac{\Vert D_A-D_B\Vert }{\Vert D_A-D_C\Vert }$$

(2)

where, d₁ and d₂ are the nearest and second nearest neighbor distances, D_A is the target descriptor, D_B and D_C are its closest two neighbors, and the symbol $\parallel •\parallel$ denotes the Euclidean distance. Lowe demonstrated that the probability of a false match (e.g., a feature with a similar pattern) significantly increases when NNDR > 0.8 [17]. Thus, matching pairs with an NNDR higher than 0.8 are not employed. Numerous studies showed that forming 1:1 matching pairs using NNDR leads to the best performance. However, matching for deformable objects the single matching-pair can be outliers, which would disrupt performance. Therefore, considering deformable-object matching, up to k candidates are selected in decreasing order of ratio, rather than selecting a single candidate with NNDR. A feature point forms 1:k matching pairs using k-NNDR. For rigid-object matching, matching pairs with a k = 1 are used, and in the deformable object-matching method, k = 2 or 3 is used.

The N × M matching pairs formed as such undergo an overlap check process, as follows:

$$ovlp\left[i,j\right]=\{\begin{array}{c} 1 \mathrm{if} M_i\left(p_i\right)=M_j\left(p_j\right),\\ 0 \mathrm{otherwise}.\end{array}\left(0\le i,j\le NM\right)$$

(3)

Matching pairs (M_k) consist of each of the feature points from the reference and query images. In Equation (3), M_k(p_k) denotes the positions of the two feature points. Here, $p_k=\left[p_k^R,p_k^Q\right]$, where $p_k^R$ is the position of the feature point extracted from the reference image, and $p_k^Q$ is the position of the feature point extracted from the query image. If $p_i^R$ is equal to $p_j^R$, or if $p_i^Q$ is equal to $p_j^Q$ when comparing the i-th matching pair (M_i) with the j-th matching pair (M_j), it is recognized as a repetition, and 1 is assigned to $ovlp\left[i,j\right]$. In this manner, 1 or 0 is assigned to all $ovlp\left[i,j\right]$, eventually generating an NM × NM overlap matrix, which has $ovlp\left[i,j\right]$ as elements. The generated overlap matrix is used for clustering during deformable-object matching.

3.2. Geometric Verification for Rigid Object-Matching

After forming matching pairs, image matching is determined through geometric verification of 1:1 matching pairs. Letting the matching pairs be (x₁,y₁),…,(x_n,y_n), the LDR set Z is obtained with the following equation:

$$z_{ij}=ln\left(\frac{\Vert x_i-x_j\Vert }{\Vert y_i-y_j\Vert }\right),Z=\{z_{ij}|i\ne j\}$$

(4)

where x_i,j is a coordinate of the reference image, and y_i,j and x_i,j are the matched-feature coordinates.

Moreover, Z denotes the LDR sets of all matching pairs. A schematic of this process is in Figure 2.

Let all features within one image follow the same bivariate normal distribution, with variance ${\mathsf{\sigma }}_x^2$ in the query image and ${\mathsf{\sigma }}_y^2$ in the reference image. Let $a^2$ be the proportion of the variances:

$$\frac{{\sigma }_x^2}{{\sigma }_y^2}=a^2$$

(5)

Then the LDR has the following probability distribution function (PDF):

$$f_Z\left(z;a\right)=2{\left(\frac{a{\mathrm{e}}^z}{{\mathrm{e}}^{2z}+a^2}\right)}^2$$

(6)

In addition, we examine the upper and lower bounds of the LDR for objects of this type. This behavior is studied by first forming LDR histogram h(k) for the matching pairs by counting the occurrences over each bin:

$$h\left(k\right)=\#(Z{{\displaystyle \cap }}^{\text{}}{\zeta }_k)$$

(7)

The bins ${\zeta }_1,\dots ,{\zeta }_K$ are adjacent intervals. The inlier behavior can be expressed by the following double inequality:

$$a\Vert x_i-y_i\Vert \le \Vert y_i-y_j\Vert \le b\Vert x_i-x_j\Vert$$

(8)

where $a$ and $b$ define the boundaries of the LDR for inliers. The LDR is restricted to an interval. The inliers would contribute to bins contained in ($-\ln b,-\ln a$) which for most cases is a limited portion of the histogram. We used the interval ($a,b),$ which is (−2.6, 2.6).

The comparison of LDR histograms between incorrectly matched images and correctly matched images is presented in Figure 3. Since a geometric transform occurs mostly in rigid object images, each matching-pair will have a similar logarithm distance and will become narrow. Exploiting this, the match between two images can be determined.

Pearson’s chi-square test is utilized to compare the correlation of h(k) and f(k). Let the LDR histogram have k bins. The histogram will be compared to the discretized model function f(k). We used these quantities to formulate the test. Equation (9) is a formula to calculate the goodness-of-fit parameter for c. A high value for c if the shape of the LDR histogram differs much from that of f(k), implied that many of the matches are inliers:

$$c={\displaystyle \sum }_{k=1}^K\frac{{\left(h_k-n{f}_k\right)}^2}{n{f}_k}\ge {\chi }_{1-a,K-1}^2$$

(9)

where n is the total number of matching pairs, and ${\chi }_{1-a,K-1}^2$ is the threshold of ${\chi }^2$ having K-1 degrees of freedom. The threshold determination method is based on an experiment by Lepsøy [9]. To obtain the false positive rate below, 1 − $a$ = 0.01 (1%), set the threshold to 70. If c is higher than the threshold, d(k) is computed using Equation (10) to find the inliers:

$$d\left(k\right)=h\left(k\right)-\mathsf{\beta }f\left(k\right),\mathrm{ }\mathsf{\beta }=\frac{{\sum }_{k=1}^{K}h\left(k\right)f\left(k\right)}{{\sum }_{k=1}^K{\left(f\left(k\right)\right)}^2}$$

(10)

where $\mathsf{\beta }$ is a weight to normalize h(k) and f(k). Figure 4a shows the comparisons of h(k) to f(k) and to d(k), respectively, between matching images and non-matching images. We can see that the LDR of matching pairs is narrow with numerous inliers, and that a match can rapidly be identified by calculating this difference.

After geometric verification based on LDR, matrix D is formed by quantizing d(k) in order to predict the number of inliers, and eigenvalue u and dominant eigenvector r are calculated using Dr = ur, used in Equation (11).

$$inlie{r}_n=1+\frac{\mathsf{\mu }}{ma{x}_{k=1,\dots ,K}d\left(k\right)}$$

(11)

Subsequently, eigenvector r is sorted in descending order, and the upper-range matching pairs corresponding to the calculated number of inliers are finally determined to be the inliers. Figure 4b shows a diagrammatic representation of this process.

Lastly, the weights of the matching pairs and the inliers are calculated, and matches are determined by calculating the ratios of the weights. Weights are based on the conditional probability of the inliers when the ratio between the NN distance and second distance during feature matching is given. Equation (12) shows the corresponding conditional probability function.

$$p\left(c|r\right)=\frac{p(r|c)}{p\left(r|c\right)+p(r|\overline{c})}$$

(12)

where c and $\overline{c}$ denote the probability of inliers and outliers, respectively, and r is the distance ratio.

3.3. Discrimination of Deformable Object Images

After generating matching pairs from the two images, x and y, and performing geometric verification, the total number of matching pairs, the number of inliers, the matching-pair weight, and the inliers’ weights are calculated using the results. These can be defined as matching information M(x,y) = {$matc{h}_n, inlie{r}_n, weight\}$. We analyzed which statistical distribution M(x,y) has, using the example images. Non-matching and deformable-object images were used as training images, and the mean, $\mathsf{\mu }$, and standard deviation, σ, of each property were calculated to form normal distribution $N(\mathsf{\mu },{\mathsf{\sigma }}^2)$. However, since the non-matching and deformable-object images show similarity in M(x,y) from numerous aspects, discrimination of the two classes is vague. Therefore, the following is proposed to find the model that minimizes the error. Analyzing numerous training images, we observed that deformable-object matching does not lead to good results if the number of inliers is less than a certain number. In other words, there must be more than a certain number of inliers. Figure 5b was obtained from experiment using more than four inliers corresponding to minimum error. Subsequently, the ratio between the number of inliers with the greatest difference in mean between the two classes, and the matching-pair weight with the least difference in variance is calculated, and t with the minimum error is obtained using a Bayesian classifier.

Let the training set be X = (x₁,c₁), (x₂,c₂),…,(x_n,c_n), x_i for the i-th properties, and let c_i be the class index of x_i, which represents non-matching (w₁) and deformable-matching (w₂). X is defined as the ratio between the inlier number from matching information and matching-pair weight, as shown in Equation (13):

$$x=\frac{inlie{r}_n}{dis{t}_n}$$

(13)

Figure 5 shows the graph obtained by separating matching information into that of incorrectly matched images and deformable matching images. Figure 5a–d shows each property of the matching information; however, it is difficult to distinguish the deformable matching pairs from non-matching pairs. Figure 5e shows the normal distribution of x for the matching pairs with more than four inliers after rigid-object matching. We can see that the distinction is clearer than in the previous graphs. Therefore, t with the minimum error was found and is used for the classifier.

However, discrimination of deformable objects does not exhibit good performance if it is based on statistical analysis only. This is because a deformable transformation cannot be defined as a certain model. Therefore, the pattern for d(k), which was used for geometric verification, was identified through machine learning-based training, and the result was used. Figure 6 shows a graphical illustration of d(k) from matching rigid-object, deformable-object, and non-matching-pair images.

Figure 6c appears to be completely different from the case above. It exhibits irregular patterns over the entire range, most of which are small in size. This implies that the LDR and the PDF are almost identical. Exploiting this characteristic, d(k) of a deformable matching-pair and d(k) of a non-matching pair were extracted from the training set, and the characteristics of each pattern were classified using support vector machine (SVM) [19], which is one of the supervised learning algorithms.

The probability of a deformable transformation is determined from the results of the previous statistical analysis and machine learning, and the deformable objects are finally discriminated through voting. The voting method combines the results obtained from various conditions, rather than from using the results of the one with the best classification performance, which is appropriate for unpredictable deformable transformations.

3.4. Deformable-Object Matching

Deformable-object images are discriminated, and finally, deformable object-matching is performed. Rigid object-matching methods mostly calculate a homograph in order to examine the geometric consistency of the features in the whole image. However, while deformable objects can have locally similar geometry, it is difficult to calculate a single homograph from the whole image. Therefore, we used a method to cluster matching pairs with similar geometric models, and we propose a method with enhanced performance through the use of clustering validation.

Letting two matching pairs be M_i, and M_j, transformations T_i and T_j and translations t_i, and t_j can be calculated from the characteristics of each matching pair, using the enhanced weak geometric consistency (WGC) [20], as shown in Equation (14).

$$\left[\begin{array}{c}{x}_q^{\prime }\\ y_q^{\prime }\end{array}\right]={\text{s}}^{\prime }\left[\begin{array}{cc}\text{cos}\mathsf{\theta }\prime & -\text{sin}\mathsf{\theta }\prime \\ \text{sin}\mathsf{\theta }\prime & \text{cos}\mathsf{\theta }\prime \end{array}\right]\left[\begin{array}{c}{x}_p\\ y_p\end{array}\right],t=\left|q^{\prime }\left(x_q,y_q\right)-q\left(x_q,y_q\right)\right|$$

(14)

where s denotes scale, θ is the dominant orientation, and t_x and t_y represent coordinate translations. Using Equation (9), the matching pairs can be expressed as $M_i=\left(\left(x_i,y_i\right),(x_i^{\prime },y_i^{\prime }),T_i\right)$ and $M_j=\left(\left(x_j,y_j\right),(x_j^{\prime },y_j^{\prime }),T_j\right),$ and the geometric similarity of the two matching pairs is calculated using Equation (15):

$$d_{geo}=\left(M_i,M_j\right)=\frac{1}{2}(\Vert X_j^{\prime }-T_i{X}_j\Vert +\Vert X_i^{\prime }-T_j{X}_i\Vert )=\frac{1}{2}\left(d_{mi}+d_{mj}\right)$$

(15)

where, $X_k={\left[x_k,y_k\right]}^t,X_k^{\prime }={\left[x_k^{\prime },y_k^{\prime }\right]}^t,k=i,j$. If transformation models $T_i$ and $T_j$ are similar, d_geo(M_i,M_j) will be close to 0. A graphical representation of this is shown in Figure 7.

Using this relationship, it can be assumed that the matching pairs with a small d_geo(M_i,M_j) have a similar geometric relation. Therefore, similarity is computed by calculating the geometric transformation between each matching pair, rather than by defining a transformation model of the whole image.

Figure 8 shows an example of an affine matrix where the geometric similarity between the matching pairs is calculated. The matrix is formed by calculating the geometric similarity between each matching pair, assuming there are 10 matching pairs. In the example, the matching pairs with high geometric similarity are the 9th and 10th matching pairs. Deformable objects are found through hierarchical clustering of matching pairs with high similarity in the calculated affine matrix.

Hierarchical clustering will group the clusters through linkages by calculating the similarity within each cluster. In order to minimize the chain effect during linkage, a k-NN method was used. While this is similar to a single-linkage method, it was extended to have k linkages, and has the advantage of robustness against the chain effect. Figure 9 shows the results of hierarchical clustering based on geometric similarity between matching pairs. Figure 9a shows the matching result when there is a geometric transformation in an image containing a rigid object. We can see from the figure that the entire object is grouped into a single large cluster due to similar geometric relations. In Figure 9b, we can see that the objects are matched separately, with each object clustered distinctly under a deformable transformation.

Hierarchical clustering eventually forms a single cluster. Therefore, clustering must be stopped at some point. During the linkage of clusters, clustering is stopped when the geometric similarity of the matching pair exceeds a set threshold. However, if such a thresholding method is the only one used, the number of clusters becomes excessive, with most of the clusters likely being false clusters. Therefore, clustering validation is used to remove false clusters. If the number of matching pairs that form the clusters is too low, it is less likely that the resulting clusters become objects. Hence, two methods are used for clustering validation. First, a cluster is determined to be a valid cluster only if the number of matching pairs that form the cluster is greater than τ_m. Secondly, a cluster is determined to be a valid cluster if the area of the matching pairs that form the cluster is larger than a certain portion of the entire area (τ_a). The area of the matching pairs that form the cluster is calculated using a convex hull. Figure 10 shows the results of removing the invalid clusters through clustering validation for each case of inlier matching and outlier matching. From inlier matching, it was observed that the clusters with a small area are removed, and for outlier matching, the accuracy was enhanced by preventing the false positives that occur due to small clusters.

4. Experiment Results

In order to evaluate the performance of the proposed matching method, the Stanford Mobile Visual Search (SMVS) dataset from Stanford University was used [21]. SMVS includes images of CDs, DVDs, books, paintings, and video clips, and is currently the standard image set for performance evaluation of image matching under MPEG-7 CDVS. The annotations of matching pairs and non-matching pairs were compiled in SMVS, through which true positives and false positives can be evaluated. Additionally, in order to evaluate deformable object-matching, performance was evaluated with varying intensities of deformation using thin plate spline (TPS) [22]. Deformation intensity was varied in three levels (light, medium, and heavy), and 4200 query images from SMVS were extended to 12,800 images. Lastly, 600 clothing images, which are natural deformable objects without artificial deformation, were collected and used. Figure 11 shows examples of SMVS images, deformed SMVS images with each level of intensity, and clothing images.

True positive rate (TPR) and false positive rate (FPR) were used to evaluate matching performance. TPR is an equation that examines robustness from among the characteristics of matching algorithms, with greater value implying better performance. In contrast, FPR examines independence from among the characteristics of matching algorithms, with a smaller value implying better performance. Moreover, for objective comparison, accuracy was used, which is defined in relation to TPR and FPR.

$$\text{TPR}=\frac{TP}{TP+FN}=\frac{TP}{P}$$

$$\text{FPR}=\frac{FP}{FP+TN}=\frac{FP}{N}$$

$$\text{Accuracy}=\frac{TP+TN}{P+N}$$

For a speed performance test, we used an Intel Xeon E3-1275 (8 core) CPU and with a clock speed of 3.5 GHz and 32 GB RAM running the Windows 7 (64-bit).

4.1. Geometric Verification Test for Rigid Object Matching

First, an NNDR higher than 0.8 was excluded in feature matching, and a Euclidean distance less than 0.6 was not used for fast calculation. This method was verified by Mikolajczyk and has been applied in most feature-matching cases [18]. Inliers were determined through LDR, while the matching score for a matching decision was calculated as follows:

$$matching score=\frac{\omega }{\left(\omega +T_m\right)}$$

(16)

where, $\omega$ is the sum of inlier distances, and T_m is the threshold value. Matching was determined for a matching score >0.5. Accordingly, the experiment was conducted by altering the T_m value. The optimum value for T_m was determined through the receiver operating characteristic (ROC) curve, and set at T_m = 3, allowing a comparison of the performances of standard matching methods. A rigid-object image from the SMVS dataset was employed as an experimental image.

Figure 12 illustrates the ROC curves of Approximate Nearest Neighbor (ANN) [23], RANSAC [7], and DISTRAT [9], which were employed in this study as rigid-object image-matching methods. A superior performance is indicated by a position closer to the top-left of the graph. DISTRAT and RANSAC exhibit similar performance in rigid-object matching. However, DISTRAT, which is based on a statistical algorithm, has the advantage of a very fast matching speed compared to RANSAC, which is based on an iteration algorithm.

4.2. Discriminating Deformable Objects Using Voting Methods

The voting method was used for discrimination of deformable object images. Two additional methods were used as conditions in each voting, as well as SVM and the statistical model described in Section 3.3. The additional methods are the matching score used in rigid-object matching, and the sum of inliers distance. Matching score employs Equation (16). A low matching score indicates a small number of inliers or insufficient weight for an image-matching determination. However, it is necessary to slightly lower the matching parameter for deformable object images. Thus, a value that is lower than 0.5, which is used in rigid-object matching, was selected, and the probability of deformable object-matching was determined. The value was set at 0.2 and employed for the experiment. Second, the ratio of the total matching-pair weight and inlier weights was calculated. This value represents the proportion of inlier weights in the total matching pairs, regardless of the number of inliers. A value of 0.28 was selected through experiment as the method for determining matching-pairs having the probability of a deformable object.

Results of the proposed discrimination process for deformable objects are shown in Figure 13. The University of Kentucky (UKY) dataset was used as the training image set [23]. The UKY dataset consists of 10,200 images of 2550 objects.

Four conditions were used for voting, and the voting value was varied throughout the experiments, where voting > 0 implies conducting deformable-object matching for all cases that have not been rigid-matched. In contrast, voting > 3 indicates no deformable-object matching in most cases.

While TPR is highest voting > 0, the execution time is inefficient, the FPR is high, and accuracy is low, since all of the numerous deformable object-matching processes were run. If the voting value increases, the TPR and accuracy decrease, overall, since the possibility of deformable objects decreases. Therefore, the best performance for average accuracy was when voting > 1.

4.3. Deformable Object-Matching Performance Test

Hierarchical clustering was used for deformable-object matching. Hierarchical clustering will create a single cluster without cutoff. Figure 14a presents experimental results obtained by changing the cutoff. As the cutoff value increases, TPR and FPR increase. Accuracy was calculated to find the optimum cutoff value. It was confirmed through experiment that the best accuracy was found at a cutoff of 30. Moreover, linkage for the clustering employed a strong k-NN in a chain effect, and an experiment was conducted to determine the optimum k. The results are presented in Figure 14b. For a higher k, the number of false positives is reduced, but complexity increases. Because there is no significant effect on accuracy when k is higher than 10, a k value equal to 10 was employed.

An experiment was conducted using the two proposed methods for clustering validation. When clusters were formed using hierarchical clustering, the number of matching pairs contained in each cluster and the area of each cluster consisting of matching pairs in the image are determined as shown in Figure 15. Each cluster can be viewed as an object in an image. When the number of matching pairs constituting an object is small, or the area is narrow, the resulting object has a low value. Hence, cluster validation was performed using this result. Figure 15a illustrates the experiment regarding false clusters when the number of matching pairs contained in a cluster has a small threshold value (τ_m). The optimum value was determined through experiment to be τ_m = 5. Figure 15b shows the calculated ratio of each cluster area and the entire image area. A false cluster was deemed to occur when the ratio (τ_a) is low. From the experiment, the best value was confirmed to be τ_a = 0.01, and hence, this value was employed.

4.4. Performance Evaluation for the Proposed Matching Method

Lastly, performance evaluation using all test images was compared to that of various matching methods. TPR, FPR, accuracy and execution time were compared, as shown in

Table 1. Performance results of image-matching methods (averages).

Methods	TPR	FPR	Accuracy	Matching Time (s)
DISTRAT [9]	83.51%	6.41%	88.55%	0.446
ANN [24]	70.27%	3.03%	83.62%	0.536
ACC [15]	86.45%	6.46%	89.99%	3.759
CDVS(Global) [10]	67.27%	0.35%	83.46%	0.003
CDVS(Local) [10]	74.94%	0.28%	87.33%	0.005
Proposed	89.78%	7.12%	91.33%	0.521

. The parameters for performance evaluation were determined though the experiments described in the previous Section 4.1, Section 4.2 and Section 4.3. The main parameters are as follows. A value of T_m = 3 for the matching score was employed for rigid-object matching. Moreover, rigid matching was deemed to occur when the calculated matching score was higher than 0.5; voting > 1, which determines the best performance, was employed for voting on discrimination of deformable object images. Finally, cutoff = 30 was employed for cluster matching in deformable-object matching, while values of τ_m = 5 and τ_a = 0.01 were employed for cluster validation. For the performance comparison, the same parameters were employed for DISTRAT [9] and ACC [15].

Compared to various matching methods, the average accuracy was highest with the proposed method, with no significant difference in execution time. DISTRAT [9] and ANN [24] are representative rigid object-matching methods, and show good results with normal images, where there is only geometric transformation rather than deformable transformation. However, a dramatic decrease in performance was seen with an increase in the intensity of deformable transformation. Moreover, while the most recent method, CDVS [10], shows extremely fast execution time, since its objective is in retrieval rather than matching (and hence, compressed descriptors are used), this method also shows a decrease in performancebecause it is intended for rigid objects. While the clustering method [15] has a TPR equivalent to the proposed method, its FPR and time complexity are high, which will present difficulties in actual applications.

Figure 16 shows a comparison of various matching methods with different image types. TPR was observed to be the best in the proposed method. While it can be seen that both TPR and accuracy with the previous methods indicate a significant decrease in performance with an increasing intensity of deformation, the proposed method shows the least amount of decrease, implying that it can be used for robust matching of any transformation.

5. Conclusions

This paper introduced a method of matching images with geometric transformation and deformable transformations using the same features. Previous methods were specialized for certain transformations, and do not show good performance with other transformations. As a solution to this issue, we proposed an adaptive image-matching method that adopts discrimination of deformable transformations. The possibility of the occurrence of a deformable transformation is discriminated using the results from statistical analysis of the matching information and supervised learning. The proposed method showed discrimination performance of more than 90% accuracy, on average. Moreover, we proposed a method of calculating the geometric similarity of each matching pair, matching the pairs by clustering those with high similarity. Previous clustering-based methods have an issue with high time complexity. However, since the use of the aforementioned method of discriminating deformable transformations leads to most non-matching images being removed, and matching concerns only images with a deformable transformation, the discrimination method brings an increase in efficiency to the overall matching method. While the proposed image-matching method exhibits a TPR of 89%, since it can generate both rigid- and deformable-matching results, it was observed that FPR does not increase, since most false positives are removed through discrimination of deformable transformations.

Future research tasks include improving the accuracy of the deformable transformation discrimination method, and developing a deformable image-matching method with a fast execution time. The accuracy of the deformable transformation discrimination method greatly influences performance. In addition to local features, which were used as a solution to the problem of accuracy, the use of global features, like color or texture, can also be considered. Since this paper intends to use single-feature detection, it is believed that the use of such additional information in actual applied systems in the future can lead to improved performance. Moreover, if it is possible to perform fast indexing that can also be used for deformable transformations, it is expected that the system can be extended to an image-retrieval system that is also robust to image transformations.