MIFT: A Moment-Based Local Feature Extraction Algorithm

Abstract

We propose a local feature descriptor based on moment. Although conventional scale invariant feature transform (SIFT)-based algorithms generally use difference of Gaussian (DoG) for feature extraction, they remain sensitive to more complicated deformations. To solve this problem, we propose MIFT, an invariant feature transform algorithm based on the modified discrete Gaussian-Hermite moment (MDGHM). Taking advantage of MDGHM’s high performance to represent image information, MIFT uses an MDGHM-based pyramid for feature extraction, which can extract more distinctive extrema than the DoG, and MDGHM-based magnitude and orientation for feature description. We compared the proposed MIFT method performance with current best practice methods for six image deformation types, and confirmed that MIFT matching accuracy was superior of other SIFT-based methods.

1. Introduction

The scale invariant feature transform (SIFT) was proposed by Lowe [1] to extract image features invariant to changes in image scale, rotation, illumination, viewpoint, and partial occlusion. SIFT has been widely used in various areas, including image stitching [2], image registration [3], object recognition [4]. Several SIFT variants and extensions have been developed recently to facilitate robust feature extraction. Ke [5] used principal component analysis (PCA) rather than histograms to reduce computational time and compared SIFT and PCA-SIFT. Bay [6] proposed speeded-up robust features (SURF) to reduce computational time. Mikolajczyk [7] presented a comparative study for several local descriptors. Kang [8] proposed a modified local discrete Gaussian-Hermite moment based SIFT (MDGHM-SIFT) that significantly improved matching accuracy by replacing gradient magnitude and orientation with accumulated MDGHM based magnitude and orientation. Junaid [9] proposed binarization of gradient orientation histograms to reduce storage and computational resources. Although these SIFT based algorithms can improve feature extraction performance, they are all the different types of Gaussian based methods and, hence, sensitive to complicated deformations, e.g., large illumination changes [10].

Gaussian–Hermite moment (GHM) has been recently shown to have merit for image features [11]. GHM base functions have different orders with different numbers of zero crossings, hence GHM can distinguish image features more efficiently, and incorporating part of the Gaussian function makes GHM less sensitive to noise. Discrete GHM (DGHM) [12,13] is a global feature representation method, and modified DGHM (MDGHM) [8] is an efficient local feature representation method generated from DGHM. Kang [14] also proposed an MDGHM-SURF descriptor that used MDGHM to extract distinctive features.

This paper proposes MIFT, an MDGHM based invariant feature transform where the MDGHM based pyramid was constructed using input image MDGHMs, in contrast to the Gaussian pyramid and DoG used in the first SIFT stage, and MDGHM based magnitude and orientation were used to calculate orientation and keypoint descriptors in the third and fourth SIFT stages rather than the original gradient method.

The remainder of this paper is organized as follows. Section 2 briefly reviews conventional SIFT and MDGHM and Section 3 describes the proposed MIFT algorithm. Section 4 presents experimental results and an example application, and Section 5 summarizes and concludes the paper.

2. Scale Invariant Feature Transform and Modified Discrete Gaussian–Hermite Moment

This section reviews the conventional SIFT algorithm [1] and MDGHM [8].

2.1. Scale Invariant Feature Transform

The SIFT is a robust local feature descriptor comprising four stages.

• Scale space extrema detection. Search keypoint candidates on the basis of the extrema over all scale images. This constructs a Gaussian pyramid and finds the local extrema in DoG images.
• Keypoint localization: Locate keypoints by removing unstable keypoint candidates having low contrast or poor localization along an edge.
• Orientation assignment. Identify the orientation for each keypoint based on local image gradient information.
• Build a descriptor for each keypoint.

2.2. Discrete Gaussian-Hermite Moment

The MDGHM has orthogonality, and calculates the moment for a local image area, acting like a filter mask to describe local features using neighboring information. Suppose I(i,j) is an image and t(u,v) is a mask with size $M\times N[0\le u\le M-1,0\le v\le N-1]$. The coordinates for t(u,v) are transformed to $-1\le x,y\le 1$ by:

$$x=\frac{2u-M+1}{M-1},y=\frac{2v-N+1}{N-1}$$

(1)

Hence, the Hermite polynomial is:

$$H_p(x)={(-1)}^p\exp (x^2)\frac{d^p}{d{t}^p}\exp (-x^2)$$

(2)

and the MDGHM mask can be expressed as:

$$\{\begin{array}{l}{\stackrel{\wedge }{H}}_p(x,\sigma )=\frac{2}{M-1}{\stackrel{\_}{H}}_p(x/\sigma )=\frac{2}{M-1}\frac{1}{\sqrt{2^{p}p!\sqrt{\pi }\sigma }}\exp (-x^2/2{\sigma }^2)H_p(x/\sigma )\\ {\stackrel{\wedge }{H}}_q(y,\sigma )=\frac{2}{N-1}{\stackrel{\_}{H}}_q(y/\sigma )=\frac{2}{N-1}\frac{1}{\sqrt{2^{q}q!\sqrt{\pi }\sigma }}\exp (-y^2/2{\sigma }^2)H_q(y/\sigma )\end{array}$$

(3)

Thus, MDGHM of an image at point $\left(i,j\right)$ can be expressed as:

$${\stackrel{\wedge }{\eta }}_{p,q}(i,j)=\frac{4}{(M-1)(N-1)}{\displaystyle \sum _{u=0}^{M-1}{\displaystyle \sum _{v=0}^{N-1}I(u+i-\raisebox{1ex}{$M$}\!\left/ \!\raisebox{-1ex}{$2$}\right.+1,v+j-\raisebox{1ex}{$N$}\!\left/ \!\raisebox{-1ex}{$2$}\right.+1)}}\times {\stackrel{\wedge }{H}}_p(x,\sigma ){\stackrel{\wedge }{H}}_q(y,\sigma )$$

(4)

Figure 1 shows MDGHM examples with three derivative orders. MDGHM acts like a Gaussian filter with multi-order derivatives. Since the base function of the nth order MDGHM changes sign n times, MDGHM can efficiently represent spatial characteristics and strongly separate image features using the multi-order derivatives. Therefore, MDGHM can be used as a filter to describe local features.

3. Proposed MIFT Algorithm

The scale invariant feature transform remains sensitive to deformations [1,8], because DoG and gradient methods do not provide distinctive information to accurately determine keypoint location in a deformed image. To obtain better matching accuracy, we propose MIFT, an MDGHM based invariant feature transform. Figure 2 shows an overview for SIFT and MIFT methods.

3.1. Stage 1

We use MDGHM based scale space to detect extrema, rather than the Gaussian pyramid and DoG used for conventional SIFT, extracting more distinctive features. The input image is down-sampled by a factor of 2 to create an octave, and MDGHM is applied to create scale images according to selected parameters for derivative order, sigma, and mask size.

Let $(i_{\alpha },j_{\alpha },s)$ be an arbitrary pixel in a scale $s$, then the local moment at $(i_{\alpha },j_{\alpha },s)$ can be expressed using MDGHM as:

$${\stackrel{\wedge }{\eta }}_{p,q}(i_{\alpha },j_{\alpha },s)=\frac{4}{(M-1)(N-1)}{\displaystyle \sum _{u=0}^{k_M-1}{\displaystyle \sum _{v=0}^{k_N-1}I(i_{\alpha }+(m_{M}u-\raisebox{1ex}{$M$}\!\left/ \!\raisebox{-1ex}{$2$}\right.+1),}}{j}_{\alpha }+(m_{N}v-\raisebox{1ex}{$N$}\!\left/ \!\raisebox{-1ex}{$2$}\right.+1),s){\stackrel{\wedge }{H}}_p(x,\sigma ){\stackrel{\wedge }{H}}_q(y,\sigma )$$

(5)

and the vertical, ${\stackrel{\wedge }{\eta }}_{p,0}(i_{\alpha },j_{\alpha },s)$ and horizontal, ${\stackrel{\wedge }{\eta }}_{0,q}(i_{\alpha },j_{\alpha },s)$, moments can be obtained for each keypoint. Therefore, we can calculate the scale space moment by summing the vertical and horizontal components:

$$\stackrel{\wedge }{L}(i_{\alpha },j_{\alpha },s)={\stackrel{\wedge }{\eta }}_{p,0}(i_{\alpha },j_{\alpha },s)+{\stackrel{\wedge }{\eta }}_{0,q}(i_{\alpha },j_{\alpha },s)$$

(6)

and more distinctive keypoint candidates can be detected using MDGHM pyramid extrema than the DoG pyramid.

Figure 3 compares and contrasts DoG and MDGHM pyramids and feature detection methods between conventional SIFT and proposed MIFT methods, and Figure 4 provides example scale space using DoG and MDGHM. Since MDGHM can include more distinctive feature information, the MDGHM pyramid can represent feature information in more detail than conventional SIFT, and building the MDGHM scale space is also somewhat simpler.

The vertical and horizontal axes in Figure 4 represent octave and scale in scale space, respectively. MDGHM scale space images extract more distinctive feature information. Points with maximum or minimum MDGHM compared with its 26 neighbors at the consecutive three scales are regarded as keypoint candidates.

3.2. Stage 2

The proposed MIFT keypoint localization is similar to conventional SIFT, except that local DoG maxima and minima are replaced by the MDGHM counterparts. We then fit the candidates to filter those with low contrast or localized along an edge.

3.3. Stage 3

Figure 5 shows the SIFT and MIFT processes to assign keypoint orientation. Conventional SIFT uses gradient magnitude and orientation, whereas MIFT uses the MDGHM magnitude, $\stackrel{\wedge }{m}(i_{\alpha },j_{\alpha },s)$, and orientation, $\stackrel{\wedge }{\mu }(i_{\alpha },j_{\alpha },s)$, for each keypoint, where:

$$\stackrel{\wedge }{m}(i_{\alpha },j_{\alpha },s)=\sqrt{{({\stackrel{\wedge }{\eta }}_{p,0}(i_{\alpha },j_{\alpha },s))}^2+{({\stackrel{\wedge }{\eta }}_{0,q}(i_{\alpha },j_{\alpha },s))}^2}$$

(7)

and:

$$\stackrel{\wedge }{u}(i_{\alpha },j_{\alpha },s)={\tan }^{-1}\frac{{\stackrel{}{\widehat{\eta }}}_{0,q}(i_{\alpha },j_{\alpha },s)}{{\stackrel{}{\widehat{\eta }}}_{p,0}(i_{\alpha },j_{\alpha },s)}$$

(8)

Each sample point around a keypoint has an MDGHM magnitude and an orientation, and the orientation histogram is calculated by summing these orientations weighted by the magnitudes. The highest histogram peak and other local peaks within 80% of the highest peak are selected as the orientations.

3.4. Stage 4

We calculate a descriptor that is invariant to deformations following a similar procedure to conventional SIFT, using MDGHM rather than gradient based magnitude and orientation, as shown in Figure 6. We calculate MDGHM magnitude and orientation for each sample point in a region around a given keypoint (Figure 6, left), and accumulate these orientations into the orientation histogram for eight directions weighted by their magnitude (Figure 6, right). Each descriptor consists of a 2 × 2 array of histograms, hence each keypoint descriptor has 2 × 2 × 8 = 32 dimensions. In this study, we used 4 × 4 × 8 = 128 dimensions to describe a keypoint.

4. Experimental Results

We performed experiments to evaluate the proposed MIFT performance using keypoint matching accuracy, compared with five SIFT relative algorithms, and considered application to ego-motion compensation for a humanoid robot.

4.1. Keypoint Matching Accuracy

Table 1. Proposed MIFT and SIFT based algorithms used for comparison.

	Alg.	SIFT	PCA-SIFT	MDGHM-SIFT	SURF	MDGHM-SURF	MIFT
Stage
1		DoG	DoG	DoG	Integral image Hessian matrix	Integral image MDGHM matrix	MDGHM pyramid
2		Localization	Localization	Localization			Localization
3		Gradient	Gradient	Accumulated MDGHM (multi-order derivatives)	Haar wavelet	MDGHM based orientation	MDGHM (single order derivative)
4		Histogram	PCA	Histogram	Sum of Haar wavelet	MDGHM based descriptor	MDGHM based histogram

Notes: Alg. = Algorithm, SIFT = scale invariant feature transform, PCA = principle components analysis, MDGHM = modified discrete Gaussian-Hermite moment, SURF = speeded up robust features, MIFT = MDGHM based invariant feature transform, DoG = difference of Gaussian.

compares the proposed MIFT method with five SIFT relative algorithms. SIFT relative algorithms generally have four stages and, hence, they are characterized by their differences.

4.1.1. Image Deformation Dataset

We conducted an experiment to matching accuracy for a dataset containing six image deformations [15]: scale, image, viewpoint, blur, JPEG-compression, and illumination. Figure 7 shows some example testing images, where the left and right images represent reference and corresponding deformed images, respectively.

4.1.2. Matching Method and Evaluation Metrics

We utilized nearest neighbor distance ratio (NNDR) matching [7] for performance evaluation, since NNDR selects only the best match. Two descriptors were considered to match if they were nearest neighbors and the distance ratio was less than a threshold. We used DR as a short descriptor for NNDR for convenience.

We used recall, 1-precision, and F-score evaluation metrics [7,14]:

$$recall=\frac{\mathrm{number}\mathrm{of}\mathrm{correct}-\mathrm{positives}}{\mathrm{total}\mathrm{number}\mathrm{of}\mathrm{positives}},$$

(9)

$$1-precision=\frac{\mathrm{number}\mathrm{of}\mathrm{false}-\mathrm{positives}}{\mathrm{total}\mathrm{number}\mathrm{of}\mathrm{matches}},$$

(10)

and:

$$F-score=2\times \frac{precision\times recall}{precision+recall},$$

(11)

where correct positive was a match for two keypoints corresponding to the same physical location, false positive was a match for two keypoints corresponding to different physical locations, and F score ϵ [0, 1].

4.1.3. Performance Evaluation

Figure 8 shows the resultant considered method performances, where each datapoint was the average for 3–6 continuously varying test images, and we increased DR 0–1 with 0.05 interval. MIFT achieved significantly superior performance than the other SIFT based algorithms for all deformations, with general performance order being MIFT, MDGHM-SIFT, SIFT, MDGHM-SURF, SURF and PCA-SIFT algorithms for most deformation cases. Thus, MDGHM had a positive effect on feature representation ability for image information.

Table 2. Performance metrics.

	Algorithm/Metric	SIFT			PCA-SIFT			MDGHM-SIFT			MDGHM-SURF			MSIFT
Deformation		Recall	Precision	F-score	Recall	Precision	F-score	Recall	Precision	F-score	Recall	Precision	F-score	Recall	Precision	F-score
Scale		0.283	0.284	0.283	0.095	0.095	0.095	0.307	0.307	0.307	0.241	0.243	0.242	0.375	0.376	0.375
Rotation		0.628	0.633	0.630	0.534	0.539	0.537	0.700	0.707	0.703	0.615	0.618	0.417	0.765	0.766	0.765
Viewpoint		0.436	0.439	0.437	0.322	0.287	0.303	0.541	0.546	0.544	0.511	0.518	0.516	0.620	0.621	0.620
Blur		0.659	0.659	0.659	0.511	0.512	0.511	0.685	0.685	0.685	0.670	0.672	0.671	0.754	0.759	0.757
Compression		0.656	0.656	0.656	0.641	0.641	0.641	0.710	0.710	0.710	0.699	0.700	0.700	0.787	0.790	0.788
Illumination		0.512	0.523	0.517	0.384	0.381	0.383	0.616	0.620	0.618	0.546	0.550	0.548	0.815	0.817	0.816

shows performance metric outcomes. MIFT F-score increased approximately 32.5%, 21.4%, 41.9%, 14.9%, 20.1%, and 57.8% for scale, rotation, viewpoint, blur, JPEG compression, and illumination distortions, respectively, compared with conventional SIFT. The proposed MIFT method exhibited significantly superior performance compared with the other SIFT based algorithms for F-score. Thus, MIFT was the most effective method tested, which attributed to employing the MDGHM based pyramid and MDGHM based feature description. The MDGHM pyramid generated more distinctive keypoints from scale space extrema detection, producing stronger histogram peaks during MIFT orientation assignment. Therefore, MIFT extracted more distinguishable final keypoints and, hence, achieved superior matching accuracy.

4.1.4. MDGHM Parameter Effects on Performance

We examined the effects due to MDGHM parameters standard deviation and derivative degree by fixing the MDGHM mask size, using F-score, as shown in Figure 9. The lower plane in each subfigure represents the SIFT F-score, the curve represents the MIFT F-score. Although the performance results exhibit some oscillation, MIFT outperformed SIFT for most parameter settings. In particular, MIFT achieved high F-scores for rotation, viewpoint, and illumination deformations for all parameter settings, with margin > 0.1 between MIFT and SIFT. For scale, blur and JPEG compression deformations, MIFT F-score was higher than SIFT for most parameter cases.

Thus, the proposed MIFT method always provided superior performance than SIFT for appropriate parameter choices. MIFT exhibited the best efficiency in terms of matching accuracy when derivative order ≈ 3–7 and sigma ≈ 0.3–0.5.

4.2. Motion Compensation Application

We applied MIFT to ego-motion compensation for a humanoid robot. Vision information obtained from a humanoid robot exhibits deformations while walking, hence compensation is mandatory to recognize the walking environment.

We first simulated an ego-motion image sequence from ideal data by including x and y axis displacement and rotation, and then calculated the apparent displacement using MIFT and SIFT, as shown in Figure 10, where the left images show the estimation, and the right images show the error. We calculated error by comparing SIFT and MIFT algorithm outcomes with ideal data. MIFT achieved significantly superior performance than SIFT, as shown in Figure 10 and summarized in

Table 3. Distorted ideal image ego-motion estimation error.

Method	Variable	Mean Error	Variance of Errors
MIFT	Rotation	0.168	0.031
	x axis	0.512	0.436
	y axis	0.509	0.411
SIFT	Rotation	0.357	0.103
	x axis	0.966	1.264
	y axis	0.976	1.218

.

Finally, we mounted an SR4000 camera on a humanoid robot and evaluated algorithm errors using real image sequences, as summarized in

Table 4. Humanoid robot [16] and camera specifications.

SR-4000 Camera		Humanoid Robot
Pixel array size	176 × 144	Height	60 cm
Modulation frequency	30 MHz	Weight	6.0 kg
Non-ambiguity range	5.0 m	Degrees of freedom	25
Field of view	44° × 35°	Sensors	8 FSRs
Dimensions	65 × 65 × 68 mm		1 six axis gyro

. Figure 11 shows a sample image sequence from the humanoid robot and Figure 12 shows the corresponding image sequence after ego-motion compensation calculated using the proposed MIFT method.

The proposed MIFT method provided significantly superior compensation and was practically useful and applicable to a real-world humanoid robot.

5. Conclusions

We proposed MIFT, an MDGHM based invariant feature transform descriptor. The SIFT-based descritpors are still sensitive to more complicated deformations because of the property of DoG used for the construction of scale-space. We proposed an MDGHM based pyramid which is less sensitive to noise and can provide more distinctive feature information than DoG, and calculated MDGHM based magnitude, orientation, and keypoint descriptors to improve the robustness of local features. We then performed experiment to compare the proposed MIFT method with various conventional SIFT approaches and parameter settings for six deformation types. The results confirmed that the proposed MIFT method provided significantly improved matching accuracy compared with conventional SIFT algorithms. We also evaluated performance effects for MDGHM parameter selections and showed that the proposed MIFT method outperformed conventional SIFT algorithms for most parameter settings. We then applied MIFT to ego-motion compensation for a humanoid robot.

However, adaptive parameter tuning for derivative orders, mask size, and MDGHM variance; and applying the proposed MIFT method to particular areas, such as image stitching and robot environment recognition, remain to be considered in future studies.