Structure-Based Feature Representation for Robust Multi-Modal Image Matching

Highlights

What are the main findings?

• The proposed MIHOG framework, which integrates local normalization, block-based keypoint detection, and max-index HOG descriptors, achieves the highest overall performance across diverse multi-modal scenarios.
• It maintains competitive geometric accuracy and high success rates under challenging conditions, including optical-SAR and rotated images, outperforming both traditional and recent state-of-the-art methods.

What are the implications of the main finding?

• By reducing intensity dependence while encoding dominant structural features, MIHOG provides a robust solution for multi-modal image matching, highlighting the effectiveness of discrete structural descriptors.
• The block-based keypoint detection and multi-scale max-index encoding offer a computationally efficient and generalizable approach, suitable for practical applications in remote sensing and sensor data analysis.

Abstract

Multi-modal image matching (MIM) remains a challenging problem due to nonlinear radiometric variations and geometric distortions across heterogeneous sensors. This paper proposes a robust feature-based matching framework that reduces reliance on intensity information while enhancing structural representation. The filter with local normalization is applied to transform the input images into a common intermediate domain. A block-based strategy is then employed to enforce a uniform spatial distribution of keypoints using the ORB (Oriented FAST and Rotated BRIEF) detector. To further suppress intensity variations and improve discriminability, a novel Max-Index-based HOG (MIHOG) is developed. This descriptor integrates multi-scale feature representations and encodes dominant structural information through discrete max-index mapping. Finally, correspondences are established using a brute-force matching strategy. Extensive experiments are conducted on two multi-modal datasets covering eight diverse scenarios. The proposed method achieves an average NCM of 224.52, RMSE of 3.4788, and SR of 92%. MIHOG obtains the highest NCM on 4/8 test scenarios and improves the average NCM by 18.3% compared with the second-best method. Meanwhile, it maintains competitive computational efficiency, with an average running time of 10.20s. These results demonstrate that MIHOG can provide dense and reliable correspondences under complex cross-modal radiometric and geometric variations.

1. Introduction

As the basic problem in computer vision and image processing, image matching plays a critical role in applications such as remote sensing, environmental monitoring, autonomous navigation, and robotic mapping. The explosion of data makes single-modal information insufficient for complex scene understanding with the rapid advancement of imaging technologies. Consequently, multi-modal data fusion has emerged as an effective paradigm for enhancing information representation. As a core component of this paradigm, multi-modal image matching (MIM) provides essential support for downstream tasks such as multi-source data fusion and semantic interpretation. It focuses on establishing reliable correspondences between images acquired from different sensing mechanisms.

However, the inherent inconsistencies across modalities make MIM remain a challenging task. These discrepancies are manifested not only in nonlinear radiometric variations but also in geometric distortions and intensity distribution differences. For instance, visible images are highly interpretable but sensitive to illumination and atmospheric conditions, whereas infrared images exhibit stronger robustness to environmental interference at the cost of lower contrast and higher noise. Although Synthetic Aperture Radar (SAR) images have all-weather imaging capabilities, they usually suffer from speckle noise and complex radiometric distortions. Such modality-specific characteristics lead to significant feature distribution discrepancies in the representation space, making it difficult to establish stable correspondences across modalities.

From the perspective of feature extraction, existing MIM methods can be roughly classified into traditional handcrafted approaches and deep learning-based methods. Traditional methods typically rely on manually designed features with certain invariance properties, offering advantages in computational efficiency and interpretability. However, their performance degrades rapidly under severe radiometric and geometric variations. In contrast, deep learning-based methods leverage data-driven feature learning to capture more discriminative representations in high-dimensional space, demonstrating superior robustness in complex scenarios. Nevertheless, these methods require large-scale annotated datasets and substantial computational resources, while often lacking interpretability.

Within traditional frameworks, area-based methods estimate correspondences between local patches to match image pairs. Common similarity metrics, such as SSD [1,2] (Sum of Squared Differences), NCC [3,4] (Normalized Cross Correlation), and MI [5,6,7] (mutual information), have been widely used to measure similarity of correspondences. For instance, Liang et al. [6] integrated spatial information with MI and employed an ant colony optimization strategy for correspondence search. Rivaz et al. [8] improved matching accuracy by introducing a context-constrained MI model, further refined with structural similarity to regularize the estimation process. Although MI exhibits better robustness to cross-modal intensity differences, these methods still fundamentally relied on pixel-level statistical characteristics. This essence of computation limited their universality in capturing robust structural information and resulted in reduced robustness under significant modality discrepancies.

Feature-based methods address these limitations by detecting salient structures (e.g., keypoints, edges) and constructing descriptors to establish correspondences. Classical methods such as SIFT (Scale-Invariant Feature Transform) and its variants have demonstrated strong performance in single-modal or weakly varying scenarios. Fan et al. [9] refined the matching strategy to improve accuracy. Zhang et al. [10] integrated the Fast Library for Approximate Nearest Neighbors (FLANN) to achieve efficient approximate nearest neighbor search in high-dimensional space. Sedaghat et al. [11] enhanced feature stability across different scenarios by incorporating orientation assignment and structural modeling mechanisms. Hong et al. [12] improved the traditional exponentially weighted average operator for SAR image registration and designed a multi-scale circular domain descriptor. However, their reliance on gradient or intensity information made them less effective in multi-modal settings, where feature repeatability and discriminability were significantly degraded. To mitigate these issues, various improvements have been proposed, including normalized filtering, phase-based representations, and structural feature modeling. In particular, structure-aware descriptors based on phase congruency (PC) have shown enhanced robustness to radiometric variations by emphasizing modality-invariant structural information. For example, Xiang et al. [13] extended the SIFT framework by extracting gradient orientation histograms from multiple image regions for visible and SAR images, thereby enhancing feature saliency. Yao et al. [14] employed a Low-Pass Butterworth Filter (LPBF) to suppress nonlinear radiometric and geometric distortions. Zhu et al. [15] constructed a coarse-to-fine matching framework that incorporated spatial constraints and Gabor-based structural features. Fan et al. [16] replaced traditional intensity information with PC and employed adaptive descriptors to encode local structural relationships. Zhang et al. [17] combined corner and patch features, introducing a weighted PC orientation model to replace gradient-based representations and improve feature reliability. Zhu et al. [18] fused multi-channel autocorrelation with logarithmic Gabor wavelet transforms to detect highly repeatable keypoints and constructed a rotation-invariant maximum index map for handling rotational variations.

Despite these advances, existing methods still face two key limitations. First, many approaches rely on local correlation operations within sliding windows, leading to high computational complexity, especially for large-scale or high-resolution images. Second, most methods focus on local feature modeling while insufficiently exploiting multi-level structural relationships, resulting in limited discriminative capability under complex cross-modal conditions. These issues highlight the need for a more efficient and robust feature representation framework that can effectively integrate structural information across different scales.

To address the aforementioned challenges, this paper proposes an enhanced feature-based matching framework tailored to multi-modal scenarios. Specifically, rather than simply assembling existing normalization, detection, and description modules, the proposed method establishes a structure-oriented matching pipeline. In this pipeline, local normalization suppresses modality-dependent intensity responses, block-based ORB (Oriented FAST [19] and Rotated BRIEF [20]) improves the spatial coverage of keypoints, and max-index Histogram of Oriented Gradients (HOG) encodes dominant structural orientation patterns in a discretized manner.

Existing structure-aware descriptors model cross-modal similarity from different perspectives; however, they often rely on relatively complex structural extraction procedures or dense local correlation operations. In contrast, this paper focuses on a lightweight structural representation by combining local normalization with max-index HOG encoding. The key distinction is that the proposed method does not seek to preserve complete continuous gradient information. Instead, it emphasizes the relative dominance of local orientation responses, thereby enhancing robustness to modality-induced intensity and contrast variations. This design differs from conventional continuous gradient encoding by prioritizing stable structural ordering rather than exact gradient magnitudes. Through the joint optimization of feature detection and description, the proposed method achieves a favorable balance among robustness, discriminability, and computational efficiency in MIM tasks.

2. Materials and Methods

Multi-modal images can be regarded as different observations of the same scene under distinct imaging mechanisms:

$$\begin{array}{ccc}\hfill I^{\left(a\right)}\left(x\right)& ={\mathcal{F}}_a\left(P\left(x\right)\right)+{ϵ}_a,I^{\left(b\right)}\left(x\right)\hfill & \hfill ={\mathcal{F}}_b\left(P\left(x\right)\right)+{ϵ}_b,\end{array}$$

(1)

where $P\left(x\right)$ denotes the underlying physical properties of the scene (e.g., reflectivity or structural characteristics); ${\mathcal{F}}_a$ and ${\mathcal{F}}_b$ denote the imaging functions of different sensors; and ${ϵ}_a$, ${ϵ}_b$ are noise terms. Since ${\mathcal{F}}_a\ne {\mathcal{F}}_b$ typically involves complex nonlinear mappings, methods based on pixel-level consistency are generally not directly applicable.

Let $f^{\left(a\right)}\left(x\right)$ and $f^{\left(b\right)}\left(y\right)$ denote the n-dimensional feature representations of images $I^{\left(a\right)}$ and $I^{\left(b\right)}$ at locations $x\in {\mathrm{\Omega }}^{\left(a\right)}$ and $y\in {\mathrm{\Omega }}^{\left(b\right)}$, respectively. Under ideal conditions, $f^{\left(a\right)}\left(x\right)\approx f^{\left(b\right)}\left(y\right)$ is expected to hold. Traditional feature descriptors often struggle to balance stability and discriminability under cross-modal settings. Structural features such as LSS and PC model intrinsic relationships via a similarity measure $\mathcal{S}$ and thus show potential for multi-modal matching. However, both f and $\mathcal{S}$ still depend on intensity distributions, making them sensitive to severe radiometric variations and local geometric distortions, which degrade matching accuracy. Therefore, reducing intensity dependence while preserving structural representation is essential. Figure 1 gives the overall framework of the proposed method. Given a multi-modal image pair, the two images are first transformed into a common intermediate representation using a local normalization filter. The images are then uniformly partitioned into blocks, and the ORB detector is applied within each block to get evenly distributed keypoints. These keypoints are then encoded into feature vectors using a max-index-based HOG (Histogram of Oriented Gradients)-like descriptor. Finally, correspondences are established via brute-force matching (BFM). Details of each stage are provided in the following sections.

2.1. Detection Based on Uniform Image Blocks

Li et al. [21] introduced a local normalization filter that transforms an image pair into a common intermediate representation, thereby emphasizing shared information across modalities:

$$I_N(x,y)=I(x,y)-{\displaystyle \frac{1}{\left|W(x,y,n)\right|}}\sum _{W(x,y,n)}I(x,y),$$

(2)

where $I_N(x,y)$ denotes the normalized image, $I(x,y)$ is the original image, $(x,y)$ represents the pixel coordinates, and $W(x,y,n)$ is a local window centered at $(x,y)$ with size $(2n+1)\times (2n+1)$. Essentially, this normalization process is equivalent to subtracting a local mean-filtered version from the original image. Since the mean filter acts as a smoothing operator that suppresses fine-grained details, the resulting normalized image preserves the dominant structural information of I, which is critical for image matching.

By introducing the integral image $I_{\mathrm{\Sigma }}$ of I:

$$I_{\mathrm{\Sigma }}(x,y)={\displaystyle \sum _{i=0}^x}{\displaystyle \sum _{j=0}^y}I(i,j),$$

(3)

the summation term in the local window $W(x,y,n)$ can be efficiently computed as

$$\begin{array}{cc}\hfill S_W(x,y)=& I_{\mathrm{\Sigma }}(x+n,y+n)-I_{\mathrm{\Sigma }}(x-n-1,y+n)\hfill \\ \hfill & -I_{\mathrm{\Sigma }}(x+n,y-n-1)+I_{\mathrm{\Sigma }}(x-n-1,y-n-1).\hfill \end{array}$$

(4)

Therefore, Equation (2) can be reformulated as

$$I_N(x,y)=I(x,y)-{\displaystyle \frac{S_W(x,y)}{{(2n+1)}^2}}.$$

(5)

For boundary pixels, the local window is adjusted according to the valid image region, and the normalization factor is set to the actual number of valid pixels.

With the above normalization, multi-modal images exhibit increased similarity, as evidenced by significantly improved MI and normalized cross-correlation (NCC) scores reported in [21]. In this work, the same preprocessing strategy is adopted for both input images.

The traditional ORB detector [22] extends FAST by incorporating orientation information, improving feature discriminability and matching performance. However, directly applying ORB still has two limitations. First, the spatial distribution of detected keypoints is often highly non-uniform. Although normalization suppresses texture to some extent, variations in gray-level distribution across regions still lead to uneven keypoint density. Second, ORB has limited capability in detecting edge features. As it is primarily designed for corner detection, reliable keypoints are difficult to extract in edge regions with a single dominant gradient and no local extrema. Moreover, the thresholding mechanism in FAST further reduces sensitivity to edge structures.

To address these issues, a block-based feature point extraction strategy is proposed to enforce a more uniform spatial distribution. Specifically, the input image is partitioned into $b\times b$ regular blocks, with each block no smaller than the minimum requirement of the ORB detector (typically $64\times 64$ pixels). ORB detection is then performed independently within each block, and at most $n_u$ keypoints with the highest responses are retained per block.

For consistent evaluation, the total number of keypoints is fixed to $N_u$, and the per-block limit is defined as $n_u=N_u/b^2$. In practice, b and $n_u$ can be adjusted according to application needs. Traditional feature-based multi-modal matching methods rely heavily on both the number and localization accuracy of keypoints. The proposed strategy is particularly suitable when the initial number of detectable keypoints cannot be predetermined, as it ensures a controlled and balanced spatial distribution.

Figure 2 illustrates an example image pair to demonstrate the effectiveness of the proposed block-based feature point extraction strategy. Urban regions with dense buildings typically contain abundant edges and corners with significant gray-level variations, whereas suburban areas, water bodies, and farmland exhibit simpler textures and smoother intensity transitions. Consequently, feature points are prone to concentrate in regions with strong intensity variations, leading to uneven spatial distribution and local clustering. This imbalance adversely affects the uniformity of feature matching and degrades accuracy, especially in large-scale or complex scenes. The proposed method improves the spatial uniformity of feature distribution, alleviates local clustering, and provides a more reliable basis for subsequent matching.

2.2. Max-Index-Based HOG

Given an input image $I(x,y)$, the HOG [23] encodes local shape information by aggregating gradient statistics over spatial regions. The gradient orientation $\theta (x,y)$ at pixel $(x,y)$ is quantized into K bins, with bin width $\mathrm{\Delta }\theta ={\displaystyle \frac{{180}^{\circ }}{K}}$ for unsigned gradients. For each spatial cell C, a histogram is constructed by accumulating weighted gradient magnitudes:

$$H_C\left(k\right)=\sum _{(x,y)\in C}g(x,y)·\omega (x,y)·\delta \left(\theta (x,y)\in \mathrm{bin}k\right),$$

(6)

where $g(x,y)$ denotes the gradient magnitude, $\omega (x,y)$ the spatial and Gaussian weighting, and $\delta (·)$ the indicator function. The number of orientation bins k is initially set to 6, and its influence on matching performance is further analyzed in Section 3.2.

To enhance robustness to illumination variations, neighboring cells are grouped into blocks, and the concatenated histogram vector h is normalized using L2-norm with hysteresis clipping:

$$h\leftarrow {\displaystyle \frac{h}{\sqrt{{\parallel h\parallel }_2^2+ϵ}}},h\leftarrow \min (h,0.2),h\leftarrow {\displaystyle \frac{h}{\sqrt{{\parallel h\parallel }_2^2+ϵ}}},$$

(7)

where ${\parallel ·\parallel }_2$ denotes the ${\mathsf{\ell }}_2$ norm and $ϵ$ is a small constant. The final descriptor is obtained by concatenating all normalized block histograms.

In this work, given a local patch p with size denoted as “patch size”, multi-scale HOG feature maps are extracted using different cell sizes, yielding ${\left\{F^{\left(m\right)}(x,y,k)\right\}}_{m=1}^3$, where m indexes scales for spatial cell C (e.g., $1\times 1$, $2\times 2$, $3\times 3$), $(x,y)$ denotes spatial coordinates, and $k\in \{1,\dots ,K\}$ is the orientation bin index. The patch size is initially set to $120\times 120$ pixels for descriptor construction, and its influence on matching performance is further analyzed in the Section 3.2. All feature maps are resized to a common resolution $H\times W$. A confidence map is defined as:

$$C^{\left(m\right)}(x,y)=\underset{k}{\max }{F}^{\left(m\right)}(x,y,k)-\underset{k}{\min }{F}^{\left(m\right)}(x,y,k),$$

(8)

which reflects the spread of orientation responses. The normalized fusion weights are:

$$w_m(x,y)={\displaystyle \frac{C^{\left(m\right)}(x,y)}{{\sum }_j{C}^{\left(j\right)}(x,y)+ϵ}},$$

(9)

and the fused feature map is:

$$F(x,y,k)={\displaystyle \sum _{m=1}^3}{w}_m(x,y)F^{\left(m\right)}(x,y,k).$$

(10)

A max-index operation is then applied to obtain a discrete structural representation:

$$R(x,y)=\arg \underset{k}{\max }F(x,y,k),$$

(11)

where $R(x,y)$ denotes the dominant orientation index. The patch is divided into $n_s\times n_s$ regions (typically $n_s=6$), and a histogram is computed for each region:

$$H_{i,j}\left(k\right)=\sum _{(x,y)\in C_{i,j}}\delta \left(R(x,y)=k\right),$$

(12)

where $C_{i,j}$ is the $(i,j)$-th cell. The final descriptor is formed by concatenating all histograms into h and normalizing:

$$h\leftarrow {\displaystyle \frac{h}{{\parallel h\parallel }_2}}.$$

(13)

This operation converts continuous magnitude-dependent responses into discrete dominance-based structural encoding. Therefore, as long as the dominant orientation bin remains stable under radiometric variations, $R(x,y)$ can remain unchanged even when the absolute response magnitudes vary. Accordingly, MIHOG does not claim strict radiometric invariance; instead, it reduces intensity dependence by combining local normalization, gradient-orientation representation, multi-scale response fusion, and dominant-index encoding.

Compared with traditional HOG, the proposed Max-Index-based HOG (MIHOG) does not directly rely on complete continuous gradient magnitudes. Instead, it first constructs a multi-scale fused HOG response tensor and then converts the continuous orientation responses into a discrete dominant-index representation. This strategy reduces the dependence on absolute intensity and response magnitudes, while preserving the dominant local structural pattern. It should be noted that MIHOG does not guarantee strict invariance to arbitrary nonlinear radiometric changes. Its robustness is mainly achieved under the condition that the dominant structural orientation remains relatively stable across modalities. Although the proposed max-index encoding shares the general idea of dominant response selection with RIFT, MIHOG differs substantially in the source and construction of the response map. RIFT builds the maximum index map from phase congruency-based orientation responses, whereas MIHOG performs dominant-index encoding on locally normalized multi-scale HOG responses. Therefore, the index value in MIHOG represents the dominant HOG orientation bin after multi-scale structural fusion, rather than the dominant phase-congruency orientation used in RIFT. Figure 3 shows the generation process of MIHOG.

3. Results

The performance of the proposed algorithm in the MIM task is evaluated using three quantitative metrics: Number of Correct Matches (NCM), Root Mean Square Error (RMSE), and Success Rate (SR). Given a known homography transformation H, a pair of keypoints is considered a correct correspondence if $\parallel H · {\mathbf{k}}_i^{\left(1\right)}-{\mathbf{k}}_j^{\left(2\right)}{\parallel }_2\le \tau$, where ${\mathbf{k}}_i^{\left(1\right)}$ and ${\mathbf{k}}_j^{\left(2\right)}$ denote keypoints in the two images (in homogeneous coordinates), and $\tau =3$ pixels. The number of correspondences satisfying this constraint is defined as NCM. RMSE measures the geometric deviation of these matches, while SR reflects the overall matching success ratio. These metrics characterize feature quality, localization accuracy, and robustness.

RMSE and SR are defined as:

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{\mathrm{NCM}}}\sum _{(i,j)\in \mathrm{NCM}}\parallel H · {\mathbf{k}}_i^{\left(1\right)}-{\mathbf{k}}_j^{\left(2\right)}{\parallel }_2^2},$$

(14)

$$\mathrm{SR}={\displaystyle \frac{{\mathbf{Q}}^{\mathrm{cor}}}{{\mathbf{Q}}^{\mathrm{all}}}}\times 100\%,$$

(15)

where ${\mathbf{Q}}^{\mathrm{cor}}$ and ${\mathbf{Q}}^{\mathrm{all}}$ denote the numbers of correctly matched and total image pairs, respectively.

For consistent evaluation, a match is regarded as a failure when NCM $<10$, in which case RMSE is set to 20 [21]. All experiments are repeated five times, and average results are reported to reduce variability. Runtime is also recorded to reflect computational efficiency.

The experiments include parameter analysis, comparative evaluation, and ablation studies. The proposed method is compared with representative approaches: DMILSS [24], sRIFD [25], RDFM [26], MSHLMO [27], LNIFT [21], CoFSM [14], MLSS [28], RIFT [29], and SIFT [30], covering handcrafted, cross-modal optimization, and deep learning-based methods.

All experiments were conducted on a laptop equipped with an Intel Core i7-12650H CPU (2.70 GHz) and 32 GB RAM, using MATLAB 2022b. For reproducibility, each method adopts the parameter settings recommended in the corresponding references. During feature detection, all methods are constrained to extract no more than 5000 keypoints per image to ensure fairness.

To provide a fair descriptor-level comparison, a unified brute-force matching strategy is adopted for all handcrafted descriptors. This setting aims to evaluate the discriminability of different feature representations under the same matching protocol, rather than to compare different matching networks or post-processing strategies. Although recent deep and hybrid matchers have demonstrated strong performance, they usually introduce additional learned priors, training data dependencies, and modality-specific adaptation effects. These factors may make it difficult to determine whether the performance gain originates from the descriptor itself or from the matcher. Therefore, the matching stage is intentionally kept simple and consistent in this work to isolate the contribution of the proposed MIHOG descriptor.

Notably, RDFM represents a hybrid paradigm that bridges handcrafted and deep learning-based approaches. It leverages features extracted from a pre-trained VGG (Visual Geometry Group) network without additional training, thereby achieving a trade-off between computational cost and cross-modal adaptability.

3.1. Datasets

Two publicly available multi-modal image datasets are employed to validate the effectiveness and robustness of the proposed method in MIM tasks. These datasets cover six representative cross-modal scenarios. The details of each dataset are described as follows.

(1)

Dataset 1:

Dataset 1 is introduced in [31] and has become a widely adopted benchmark for MIM. It includes six types of image pairs: optical–optical, optical–infrared, optical–SAR, optical–depth, optical–map, and nighttime. Each category contains 200 image pairs, all exhibiting varying degrees of nonlinear radiometric differences. It should be noted that the optical–depth and nighttime image pairs were acquired using a Microsoft Kinect v2 sensor (Microsoft Corporation, Redmond, WA, USA) and a binocular imaging system (comprising a visible camera and a thermal infrared camera) across 26 different scene locations, respectively. Strictly speaking, these data do not fall within the category of remote sensing imagery. Nevertheless, they are included in this study to further evaluate the generalization capability of the proposed method across heterogeneous sensing modalities.

To further increase task complexity, random rotation transformations are applied to all image types except optical–map, with rotation angles ranging from $[0^{\circ },{90}^{\circ })$. In addition, optical–optical and optical–SAR pairs are augmented with scale variations in the range of $[0.5,2)$. These controlled perturbations introduce compounded challenges, including radiometric inconsistency, rotational variation, and scale changes, thereby enabling a more comprehensive evaluation of algorithm robustness.

Fifty image pairs are randomly selected from each of the six categories, resulting in a total of 300 test pairs for performance evaluation.

(2)

Dataset 2:

Dataset 2 is derived from the OSdataset [32], an optical-SAR dataset that contains 2673 image pairs of size $512\times 512$ or 10,692 pairs of size $256\times 256$. The SAR images were acquired by the Gaofen-3 satellite (China Academy of Space Technology, Beijing, China) operating in C-band Spotlight mode, while the corresponding visible images were sourced from the Google Earth platform. This dataset covers more than 15 urban regions worldwide, including Beijing, Wuhan, Rennes, Omaha, and Dwarka.

To ensure data quality, all image pairs are manually screened to remove blurred samples and those with significant scene inconsistencies or unrecognizable object changes. Additionally, SAR image intensities are enhanced to improve visual clarity and interpretability.

One hundred optical-SAR image pairs with a resolution of $512\times 512$ are randomly selected from the dataset to form Dataset 2 for performance evaluation. In addition, random rotation transformations within the range of $[0^{\circ },{180}^{\circ })$ are manually applied to the SAR images in each pair, further increasing the difficulty of matching.

Figure 4 presents representative examples for each image type.

3.2. Parameter Selection

This section involves two key parameters: the patch size p and the number of orientation bins k. To jointly evaluate matching correctness and geometric accuracy, a composite metric is constructed in a normalized space. Specifically, NCM and RMSE are first standardized and subsequently normalized to eliminate scale differences and ensure comparable contributions.

Based on the normalized variables, the objective function is defined in a logarithmic form to balance the influence of different factors:

$$\mathcal{J}\left(p\right)=\widetilde{\mathrm{NCM}}{\left(p\right)}^{\lambda }·\exp \left(-(1-\lambda )\widetilde{\mathrm{RMSE}}\left(p\right)\right)·\exp (-\gamma \tilde{p}),$$

(16)

where $\widetilde{\mathrm{NCM}}$, $\widetilde{\mathrm{RMSE}}$, and $\tilde{p}$ denote the normalized NCM, RMSE, and patch size, respectively. The parameter $\lambda \in [0,1]$ controls the relative importance between matching correctness and geometric accuracy and is set to $\lambda =0.5$ to ensure equal contribution. The parameter $\gamma$ regulates the penalty on the patch size, preventing excessively large support regions.

Compared with direct linear combinations, the proposed formulation operates in a unified scale and adopts a multiplicative structure, which improves numerical stability and interpretability. The exponential penalty ensures that larger patch sizes are progressively suppressed, while the power-law term preserves the discriminative contribution of matching correctness. This design enables a principled trade-off among matching capability, geometric precision, and computational efficiency, leading to a stable and well-defined optimal patch size.

Furthermore, the number of correct matches obtained per unit time, denoted as the ${\mathcal{U}}_{\mathrm{NCM}}$, is introduced to evaluate the trade-off between computational efficiency and matching performance.

Table 1. Evaluation results for the patch size p with fixed $k=6$.

Patch Size p
	72	84	96	108	120	132	144
RMSE ↓	4.6185	4.3206	4.0543	3.4773	3.4788	3.6141	3.8333
NCM ↑	146.4975	175.4200	196.3675	211.5525	224.5175	234.7775	240.1525
SR ↑	85.50%	87.25%	88.75%	92.00%	92.00%	91.25%	90.00%
$\mathcal{J}$ ↑	0.0006	0.3251	0.4061	0.5055	0.4683	0.3974	0.3147
${\mathcal{U}}_{\mathrm{NCM}}$ ↑	20.9839	21.3331	21.1270	20.9237	22.7586	21.0133	18.3025

“↑” indicates that a larger value is better, while “↓” indicates that a smaller value is better. The best values are shown in bold, and the second-best values are underlined.

and

Table 2. Evaluation results for the number of orientation bins k with fixed $p=120$.

	Orientation Bins k
	4	5	6	7	8	9
RMSE ↓	3.7961	4.0178	3.4788	3.7962	3.8429	3.9781
NCM ↑	220.7975	225.0850	224.5175	220.6725	217.3325	213.3375
SR ↑	90.25%	89.00%	92.00%	90.25%	90.00%	89.25%
$\mathcal{J}$ ↑	0.3048	0.3114	0.5009	0.3022	0.2136	0.0003
${\mathcal{U}}_{\mathrm{NCM}}$ ↑	20.1660	19.0072	22.7586	19.3260	22.0738	20.6224

“↑” indicates that a larger value is better, while “↓” indicates that a smaller value is better. The best values are shown in bold, and the second-best values are underlined.

present the evaluation results under different configurations of parameters p and k. It is worth noting that the results for $p=108$ and $p=120$ in Table 1 are very similar. In this case, priority is given to NCM and ${\mathcal{U}}_{\mathrm{NCM}}$ for selection. Based on this criterion, $p=120$ and $k=6$ are ultimately chosen as the optimal parameter configuration for the subsequent experiments.

3.3. Ablation Experiments

In this section, five ablation experiments are conducted to systematically analyze the contribution of each module in the proposed framework. The Baseline uses the original ORB detector and the original HOG descriptor. M1 replaces the original ORB detector with the proposed block-based uniform ORB detector while keeping the original HOG descriptor unchanged, thereby evaluating the effect of Uniform Blocks. M2 introduces the multi-scale HOG Fusion Map without Max-Index encoding. M3 introduces Max-Index encoding to form a Max-Index HOG descriptor without the fusion-guided design. The final MIHOG integrates all proposed components. The corresponding configurations and results are summarized in

Table 3. Ablation results of different module combinations.

Modules	Uniform Blocks	Fusion Map	Max-Index	NCM ↑	RMSE ↓	SR ↑
Baseline				41.2052	10.8971	31.25%
M1	✔			137.6674	9.1972	48.50%
M2	✔	✔		63.3455	10.3727	41.75%
M3	✔		✔	179.9861	4.0371	68.25%
MIHOG	✔	✔	✔	224.5175	3.4788	92.00%

“↑” indicates that a larger value is better, while “↓” indicates that a smaller value is better. The best values are shown in bold.

. The effectiveness of each component is clearly demonstrated through these ablation studies.

Among all modules, the max-index encoding plays a pivotal role in enhancing cross-modal adaptability, indicating that reducing reliance on intensity information is critical for MIM tasks. The fusion feature map primarily strengthens the representation of local structural information, while the block-based partitioning strategy promotes more reliable and accurate keypoint detection through spatial regularization.

Notably, M1 outperforms M2 because the Fusion Map module in M2 introduces dense aggregation, which weakens feature discriminability and amplifies ambiguous responses in the absence of an effective selection mechanism. Without the max-index constraint, this fusion process degrades matching reliability rather than improving it.

To further clarify the coupling effect between the Fusion Map and Max-Index modules, local response maps are visualized in Figure 5. The Fusion Map aggregates multi-scale structural responses and thus produces denser activations around edges and textured regions. However, these dense responses also include weak and redundant activations, which may blur the distinction between corresponding and non-corresponding local patches when directly used for descriptor construction. In contrast, the Max-Index operation selects the dominant response channel at each local position, converting continuous magnitude-dependent responses into discrete dominance-based structural encoding. As shown in Figure 5, this operation suppresses non-dominant responses while preserving clearer structural patterns. This explains why using the Fusion Map alone degrades performance, whereas combining it with Max-Index yields consistent performance gains.

The integration of these components enables MIHOG to achieve a well-balanced trade-off between matching accuracy and robustness in MIM.

3.4. Comparative Experiment

3.4.1. Qualitative Results

Figure 6 presents the matching results of different methods across all datasets. It can be observed that MIHOG consistently demonstrates stable and superior matching performance. Although it does not achieve the best results in certain scenarios, such as optical–optical and optical–depth, its ability to attain a 100% success rate across all test samples highlights the effectiveness of its intensity-invariant design for multi-modal matching.

Figure 7 illustrates residual maps for evaluating alignment accuracy by measuring pixel-wise intensity (or structural) differences after geometric transformation. “w/o R” and “w/ R” denote the settings without rotation and with rotation, respectively. The panels marked as “Fail” indicate that the corresponding method failed to produce valid matching or registration results for the given image pair.

Specifically, one image is warped into the coordinate system of the other using the estimated transformation, and the absolute difference between the aligned images is computed. To ensure a fair evaluation, residuals are calculated only within the valid overlapping region, excluding padded areas introduced during warping. For visualization, a percentile-based normalization is applied to suppress outliers and enhance contrast. The mean residual within the valid region serves as a quantitative indicator, where lower values correspond to better alignment quality.

3.4.2. Quantitative Results

The quantitative results across the three metrics demonstrate consistent performance trends under different modality combinations. In terms of NCM result (the average NCM is indicated on the top of each bar in Figure 8), MIHOG achieves the highest average number of correct matches (224.52), showing clear advantages especially in challenging scenarios such as optical-SAR (Dataset 2, w/o R) and optical-map, where it significantly outperforms the competing methods. DMILSS and RDFM obtain moderate results, while traditional methods such as SIFT and MLSS yield substantially fewer correct correspondences, indicating limited robustness under cross-modal variations.

For RMSE (

Table 4. RMSE results for all methods.

	Optical–Optical	Optical–Infrared	Optical–SAR	Optical–Map	Optical–SAR (Dataset 2) w/o R	Optical–SAR (Dataset 2) w/ R	Nighttime	Optical–Depth	AVE
MIHOG (2026)	2.7589	2.0635	2.7572	2.4117	2.0294	3.4878	3.8757	8.4464	3.4788
DMILSS (2025) [24]	3.1256	2.4176	6.0152	2.3890	2.0456	5.6391	3.5112	17.4527	5.3245
sRIFD (2023) [25]	1.9868	2.0141	2.2683	1.9677	1.9770	1.9647	2.0115	1.9840	2.0217
RDFM (2023) [26]	2.0236	6.5850	2.2846	2.0221	2.0597	4.4723	2.7484	10.7748	4.1213
MSHLMO (2022) [27]	2.3899	16.7348	9.9102	6.6957	2.0729	4.9080	9.1650	9.5296	7.6758
LNIFT (2022) [21]	7.7805	5.2677	7.4360	2.7759	2.7980	6.6810	2.4328	6.7032	5.2344
CoFSM (2022) [14]	3.8771	20	15.9486	19.2720	11.6859	17.8177	15.6423	15.2538	14.9372
MLSS (2022) [28]	6.3285	4.5582	9.5450	13.1544	8.4754	18.5593	7.0720	19.6390	10.9165
RIFT (2020) [29]	6.3248	8.8393	12.4525	3.0708	8.1520	12.8169	4.5371	9.5388	8.2165
SIFT (2004) [30]	6.1496	20	20	20	20	20	19.6303	19.2575	18.1297

The best values are shown in bold, and the second-best values are underlined.

), sRIFD achieves the lowest average error (2.0217), reflecting highly stable geometric alignment across most datasets, followed by MIHOG with a competitive average RMSE of 3.4788. In contrast, methods such as CoFSM, MLSS, and RIFT exhibit noticeably larger errors, particularly under severe modality differences, suggesting weaker geometric consistency.

Regarding SR (

Table 5. SR results for all methods.

	Optical–Optical	Optical–Infrared	Optical–SAR	Optical–Map	Optical–SAR (Dataset 2) w/o R	Optical–SAR (Dataset 2) w/ R	Nighttime	Optical–Depth	AVE
MIHOG (2026)	96%	100%	96%	90%	98%	64%	100%	92%	92%
DMILSS (2025) [24]	94%	98%	78%	92%	98%	14%	100%	80%	81.75%
sRIFD (2023) [25]	100%	100%	98%	100%	100%	100%	100%	100%	99.75%
RDFM (2023) [26]	98%	72%	96%	94%	100%	50%	100%	84%	86.75%
MSHLMO (2022) [27]	98%	18%	56%	60%	74%	58%	100%	84%	68.50%
LNIFT (2022) [21]	68%	82%	70%	98%	96%	74%	96%	74%	82.25%
CoFSM (2022) [14]	88%	0%	22%	24%	4%	26%	46%	12%	27.75%
MLSS (2022) [28]	76%	86%	58%	72%	38%	2%	64%	8%	50.50%
RIFT (2020) [29]	76%	62%	42%	86%	94%	58%	66%	40%	65.50%
SIFT (2004) [30]	76%	0%	0%	2%	0%	4%	0%	0%	10.25%

The best values are shown in bold, and the second-best values are underlined.

), sRIFD attains the highest overall success rate (99.75%), achieving nearly perfect performance across all scenarios. MIHOG ranks second with an average SR of 92%, maintaining high success rates in most cases but showing some degradation in more challenging conditions. Other methods, including RDFM and LNIFT, present moderate success rates, whereas CoFSM, MLSS, and SIFT demonstrate significantly lower reliability.

For the running time results (

Table 6. Running time results for all methods.

	Optical–Optical	Optical–Infrared	Optical–SAR	Optical–Map	Optical–SAR (Dataset 2) w/o R	Optical–SAR (Dataset 2) w/ R	Nighttime	Optical–Depth	AVE
MIHOG (2026)	17.01	8.79	8.36	10.75	10.33	10.90	8.08	7.33	10.20
DMILSS (2025) [24]	193.96	206.19	183.91	9.44	16.74	230.28	145.47	142.73	141.09
sRIFD (2023) [25]	20.09	13.51	11.65	16.17	77.62	20.47	14.07	16.96	23.82
RDFM (2023) [26]	5.77	3.13	3.05	4.16	6.27	7.14	3.08	6.02	4.83
MSHLMO (2022) [27]	121.40	73.36	34.73	111.55	277.85	109.36	37.36	54.92	102.57
LNIFT (2022) [21]	0.72	0.63	0.64	0.66	0.72	0.81	0.62	0.66	0.68
CoFSM (2022) [14]	13.14	1.75	3.88	4.56	10.44	18.71	2.08	3.88	7.31
MLSS (2022) [28]	141.45	24.39	41.74	45.79	59.44	101.68	24.98	48.48	60.99
RIFT (2020) [29]	40.60	28.78	45.62	24.30	34.93	56.28	51.78	136.79	52.38
SIFT (2004) [30]	0.36	0.05	0.10	0.14	0.27	0.34	0.07	0.24	0.20

The best values are shown in bold, and the second-best values are underlined.

), SIFT and LNIFT achieve the lowest average runtimes of 0.20 s and 0.68 s, respectively. However, their computational advantages are accompanied by inferior matching performance, particularly under severe cross-modal variations. Among the methods with competitive matching capability, MIHOG demonstrates relatively high efficiency with an average runtime of 10.20 s, substantially lower than those of DMILSS (141.09 s), MSHLMO (102.57 s), MLSS (60.99 s), and RIFT (52.38 s). Although RDFM and CoFSM are slightly faster, they generally yield lower matching accuracy and robustness.

4. Discussion

The experimental results reveal several important characteristics of different methods under multi-modal conditions. First, there exists a clear trade-off between the number of matches and geometric accuracy. As shown in the quantitative results, MIHOG achieves the highest NCM on most datasets, indicating its strong capability to establish a large number of correspondences. However, its RMSE is slightly higher than that of sRIFD, suggesting that increasing the number of matches does not necessarily guarantee optimal geometric precision. In contrast, sRIFD consistently yields lower RMSE and near-perfect SR across datasets, reflecting a more conservative matching strategy that prioritizes geometric stability, even at the cost of fewer correspondences. sRIFD constructs multiple descriptors without the computing main orientations and only keeps the best feature vector, which is why it can get high accuracy. The max-index strategy of MIHOG improves robustness by suppressing unstable magnitude variations, but it also discards secondary orientation responses. Therefore, it is more suitable for regions with clear dominant structures than for weak-texture or repetitive-pattern regions.

The performance variation across different modalities highlights the intrinsic difficulty of cross-modal matching. In relatively consistent scenarios, such as optical–optical and optical–infrared, most methods achieve satisfactory results. However, in more challenging cases, such as optical–SAR, optical–map, and nighttime imagery, performance degrades significantly. This is reflected by reduced NCM and SR, along with increased RMSE. In particular, traditional gradient-based methods (e.g., SIFT) exhibit pronounced performance deterioration in these scenarios, indicating their strong reliance on intensity consistency and limited adaptability to nonlinear radiometric variations. Although RDFM combines the VGG network, the pre-trained model is insufficient to deal with different transformations of data and is especially sensitive to geometric transformations. However, when there is no obvious geometric transformation in the image, the advantage of RDFM is extremely obvious, and the single NCM is even higher than 1000, which is remarkable for an untrained deep learning-based method.

Computational efficiency is another important factor in multi-modal matching. While SIFT and LNIFT achieve the shortest running times, their matching performance deteriorates significantly under severe modality differences. In contrast, several high-performing methods, including DMILSS, MSHLMO, and RIFT, incur substantially higher computational costs. MIHOG achieves a favorable trade-off between efficiency and robustness, requiring only 10.20 s on average while consistently delivering strong matching performance across diverse scenarios.

The similarity between MIHOG and RIFT lies in the use of dominant-index encoding to reduce sensitivity to unstable response magnitudes. However, their technical routes are different. RIFT relies on phase congruency to construct radiation-invariant structural maps, while MIHOG uses local normalization and multi-scale HOG fusion to construct a lightweight structural response representation. Thus, MIHOG provides a computationally simpler and detection-integrated alternative for multi-modal matching.

Geometric transformations, particularly rotation, further amplify the robustness differences among methods. As observed from both the matching results and residual maps, the performance of many approaches degrades significantly in the presence of rotation, as evidenced by decreased SR and increased residual errors. In contrast, MIHOG and sRIFD maintain relatively stable performance under such conditions, indicating stronger resilience to geometric distortions. This observation is consistent with the residual map analysis, where more accurate registrations correspond to lower residual intensity and a more uniform error distribution. However, the max-index discretization also introduces an inherent information-loss boundary. Since only the dominant orientation index is retained, secondary gradient responses and subtle magnitude variations are suppressed. This design is beneficial when cross-modal intensity differences make exact gradient magnitudes unreliable, but it may reduce discriminability in repetitive textures, weak-edge regions, or scenes where fine-grained orientation differences are essential. This explains why MIHOG can obtain a larger number of correct matches in most datasets, whereas methods such as sRIFD may achieve lower RMSE in some cases by adopting a more conservative and geometrically stable description strategy.

Finally, qualitative results and residual visualizations provide important complements to quantitative metrics. While NCM, RMSE, and SR offer an overall assessment of performance, residual maps intuitively reveal the spatial distribution of errors and highlight locally misaligned regions—information that is difficult to capture through scalar metrics alone. This combined quantitative and qualitative analysis demonstrates that methods with similar numerical performance may still exhibit notable differences in spatial registration quality.

Compared with existing structure-aware methods, MIHOG follows a different structural encoding strategy. PC-based and RIFT-like methods mainly rely on phase congruency or radiation-insensitive orientation maps to extract modality-invariant structural cues. These representations are effective for suppressing nonlinear intensity differences, but their performance depends strongly on the quality of structural map construction and local correlation-based description. CoFSM and LNIFT improve SIFT-like descriptors by incorporating structural or normalization-based information, but they still largely follow the paradigm of continuous local feature representation. In contrast, MIHOG uses local normalization to construct a more comparable intermediate representation and then applies max-index HOG to encode the dominant orientation response in each local region. Therefore, its robustness comes from preserving structural dominance relationships rather than retaining complete continuous gradient distributions.

This distinction can also be observed when comparing MIHOG with classic structural descriptors such as CFOG [33], HOPC [34], MOTIF [35], and HAPCG [36]. CFOG describes local structures using dense gradient-orientation responses, while HOPC and HAPCG exploit phase-congruency-related information to improve robustness to nonlinear radiometric differences. MOTIF emphasizes structural pattern representation for cross-modal matching. Although these descriptors have played important roles in multi-modal image matching, they generally preserve continuous gradient, phase-congruency, or structural response distributions for descriptor construction. By contrast, MIHOG first constructs locally normalized multi-scale HOG response maps and then converts them into a discrete dominant-index representation through max-index encoding. As a result, MIHOG emphasizes the relative dominance of local structural patterns rather than the complete preservation of continuous response magnitudes. In addition, the block-based uniform ORB detector improves the spatial distribution of keypoints, further distinguishing MIHOG from descriptors that mainly focus on local feature description alone.

Robust multi-modal matching requires a careful balance among the number of correspondences, geometric accuracy, and robustness to both radiometric and geometric variations. The experimental results indicate that emphasizing structural consistency and geometric invariance is more effective for achieving stable performance in complex multi-modal scenarios, whereas approaches heavily reliant on intensity information remain limited under severe modality discrepancies.

5. Conclusions

This paper presents a robust feature-based framework for MIM, with a focus on reducing intensity dependence and enhancing structural representation. By combining local normalization, block-based keypoint detection, and the proposed MIHOG, the method effectively addresses the challenges posed by nonlinear radiometric variations and geometric distortions. The max-index encoding mechanism plays a crucial role in improving cross-modal adaptability by transforming continuous gradient responses into discrete structural representations.

Experimental results on diverse multi-modal datasets demonstrate that the proposed method achieves superior performance in terms of matching quantity and competitive geometric accuracy. Compared with existing approaches, it provides a better trade-off between robustness and precision, particularly in challenging scenarios such as optical-SAR and nighttime images. Additionally, the block-based strategy improves the spatial distribution of keypoints, contributing to more stable matching results.

The proposed method offers an effective and computationally efficient solution for MIM. Future work may focus on further improving geometric accuracy and extending the framework to handle more complex transformations or integrating learning-based strategies for enhanced adaptability.

It should also be acknowledged that MIHOG is still built upon a handcrafted detection-description framework. Therefore, it cannot completely avoid the inherent limitations of traditional feature matching methods, such as reduced robustness in weak-texture regions, large scale variations, and strong noise interference. Compared with learning-based methods, handcrafted descriptors are also less capable of modeling high-level semantic correspondences and complex cross-modal relationships. However, a learning-free framework remains valuable in multi-modal remote sensing scenarios, where large-scale accurately aligned training data are often difficult to obtain and the imaging mechanisms vary significantly across sensors. In such cases, training-free descriptors offer advantages in interpretability, low training cost, and cross-dataset applicability. The proposed MIHOG aims to improve this type of framework by combining local normalization, spatially uniform keypoint detection, multi-scale HOG response fusion, and max-index dominant structural encoding. Nevertheless, integrating the proposed structural encoding strategy with trainable feature learning models is a promising direction for future work.