A Topology Based Automatic Registration Method for Infrared and Polarized Coupled Imaging

Abstract

In multi-source camera collaborative imaging research, it is known that the differences in size and resolution of the sensor chip, the angle of view and field of view when imaging, and the imaging characteristics of optical systems between cameras, makes image registration a topic that can never be avoided in data analysis and post-processing. Additionally, lacking common features between multi-source images means that the accurate registration of multi-modal images can only be completed manually. Aiming at the registration problem of the polarization parameter image and infrared image, this study takes advantage of the invariant feature of the imaging target topology and introduces the image texture-based segmentation method to obtain the target topology structure. Subsequently, the registration control points are extracted based on the target topology skeleton, which can break through the limitation of feature differences, improve the robustness of the algorithm to target transformation, and realize the automatic registration of multi-source images.

1. Introduction

Integration of infrared messages and polarization messages will significantly increase the amount of information in the image and enhance the ability of target detection and recognition [1,2,3,4,5,6]. However, the coupled infrared camera and polarized camera imaging in different perspectives, fields of view, and sizes [7] introduce misalignment in the resulting images, which is a common problem in multi-sensor detection methods. Before further analyses, the images need to be aligned properly, which is difficult to implement in hardware. Therefore, it is certainly more practical and economical to seek an available registration algorithm in software [8].

The image registration algorithm aligns two or more images of the same scene with matching contents. The existing registration methods are commonly classified into intensity-based and feature-based methods. Intensity-based methods include metrics based such as phase correlation [9], Mutual Information [10,11], and optimization based methods such as BSO (brain storm optimization) [12], GA (genetic algorithms) [13], PSO (particle swarm optimization) [14], Powell [15], etc. Intensity-based image registration methods require consistent gray-level information of images, therefore working well in monomodal images taken by single-modal cameras. By contrast, intensity-based methods have a limited performance in registering multi-modal images from heterogenous imaging systems, and usually suffer from image distortions, illumination changes and heavy computational complexities. It can be inferred that such algorithms are difficult to be applied in infrared and visibly polarized images with differences in grayscale distribution and content structure.

Feature-based methods are the most commonly used methods in practical applications [16], their representative algorithms are SURF (speeded-up robust feature) [17], Harris [18], FAST [19], BRISK (binary robust invariant scalable keypoints) [20], MSER (maximally stable extremal regions) [21] and the newly proposed MIND [22] (modality independent neighborhood descriptor), etc. This kind of algorithm is conducted based on descriptors related to structures such as points, corners and edges. As for heterogeneous image registration, the differences in sensitive information between heterogeneous sensors makes it extremely difficult to accurately catch the common features in both images. In the polarization and infrared image registration, infrared images are characterized by smooth intensity regions with different gray levels, while polarization parameter images have less gray level information, mainly contour and texture information. It is difficult to register the above two images with only one feature-based algorithm or intensity-based algorithm.

To overcome the lack of common features, we addressed the topological structure in the heterogeneous images, and implemented image segmentation and morphology processes in this work. Our Segment-Skeletonize-Registration (SSR) segments the input images into regions with distinct geometric features, skeletonizes the regions into skeleton branches, and finally, takes the endpoints of the skeleton as registration control points. In addition, the polarization parameter images were segmented into regions according to the texture information, and the gray level was assigned according to different regions. Thereafter, the polarization parameter image features can be transformed into features common with the infrared image. Instead of registering the images by gray level regions directly, the SSR algorithm takes advantage of the topological characteristics of the target to process the segmented region morphologically, and to obtain more robust feature points.

The remainder of this paper is organized as follows: Section 2 presents the principle of our methodology in detail; Section 3 presents the procedure of the experiment, including the acquisition of images and the implementation of the algorithm, and describes the results of SSR and the comparison with other algorithms; and finally, the conclusion is drawn in Section 4.

2. Methodology

This section introduces key principles involved in SSR, and how they are adjusted to cooperate in the registration.

2.1. Overview of the Proposed Algorithm

SSR transforms the extraction of common points into the extraction of common objects, which is different from the traditional registration algorithm. Firstly, regular preprocess, such as the median filter, is sufficient for de-noising. Depending on the imaging conditions, a top-hat transform might be required to remove the halo in infrared images. Then, to extract objects from infrared and polarization images which vary widely in image style, different segmentation methods are chosen, that is, a threshold-based algorithm for the infrared image target segment and a texture-related method for polarization parameter image target segmentation. Subsequently, morphological skeletonization is performed in segmented images to obtain the topological endpoints of the target as registration control points for later matrix transformation. The algorithm flowchart is shown in Figure 1.

2.2. Target Segmentation from Pending Images

To obtain a common target from the two images, we processed image segmentation. The segmentation for the infrared image is relatively easy. Because the infrared image itself is characterized by gray levels, in most cases, a reasonable threshold value such as the Otsu threshold method [23] can achieve the binary classification of the target and the background. The polarization parameter image is too rich in texture and lacks gray-level information. In polarized parametric images, such as the degree of polarization image and the angle of polarization image, the target and the background usually exhibit vastly different textures. Sorting out polarized texture regions to match gray-scale regions within infrared images is our goal of image segmentation, therefore, a texture-sensitive segmentation is needed.

The Gabor filter is constructed by a sinusoidal function and Gauss function, so it is sensitive to periodic information. Since texture features can be considered as a combination of periodic signals of different angles and frequencies, we used a filter bank with multiple Gabor filters in different angles to filter the polarization parameter images for sub-band images. Figure 2 is a graphical representation of the Gabor filter bank. We took 4 angles evenly at 180 degrees, and the frequency difference was achieved by scale differences in filters.

After filtering, the texture part in the image will be extracted and the smooth part will be removed. The texture of different directions will be divided by the filter of corresponding angles to realize the binary classification of each sub-band image. In the polarized parameter image, the region of interest in the figure appears as dense periodic stripe. To process the stripe into connected regions, an active contour method [24] is conducted to segment the result. It placed an initial circle on the image and changed the contour of the circle with the optimization process, the boundary achieved when

$$\min \left({\mathrm{Q}}_{\mathrm{in}}-\overline{{\mathrm{Q}}_{\mathrm{in}}}\right)+\left({\mathrm{Q}}_{\mathrm{out}}-\overline{{\mathrm{Q}}_{\mathrm{out}}}\right)$$

(1)

where ${\mathrm{Q}}_{\mathrm{in}}$ represents the gray value of each pixel inside the contour, $\overline{{\mathrm{Q}}_{\mathrm{in}}}$ represents the average gray value of each pixel inside the contour, ${\mathrm{Q}}_{\mathrm{out}}$ represents the gray value of each pixel outside the contour, and $\overline{{\mathrm{Q}}_{\mathrm{out}}}$ represents the average gray value of each pixel outside the contour. The minimization was conducted by the gradient descend method. When the minimum value is obtained, the gray values inside and outside the boundary tended to be consistent with their average values. It was considered that the accurate contour was drawn, and finally, the polarization image target and background were segmented.

2.3. Region Skeletonization for Control Points

The binary images of polarization parameter image and infrared image were obtained after the segmentation we introduced in Section 2.2.

Considering that imaging is a mapping from a real target to a digital signal, the topological structure of the target does not change with the imaging modality. In our case, based on the characters mentioned above, we put forward that the endpoint of the morphology skeleton could be the registration basis in this study. Image skeletonization is the ultimate of image thinning, which is topology-preserving. It reduces the elongated parts to one-pixel-thick arcs or closed curves, without changing the connectivity properties of the object or its background [25]. It is widely used in gait recognition, motion tracking and fingerprint analysis [26,27,28,29]. Taking the topological structure of the target as the registration basis can guarantee that the registration basis of the heterogeneous images remains invariable when the imaging sensor changes. We skeletonized the segmented binary images of infrared and polarization parameter simultaneously. Image skeletonization is represented by erosion and opening operation, and the formula [30] is as follows:

$$\mathrm{S}\left({\mathrm{Q}}_{\mathrm{in}}\right)={\displaystyle \bigcup }_{\mathrm{k}=0}^{\mathrm{K}}{\mathrm{S}}_{\mathrm{k}}\left({\mathrm{Q}}_{\mathrm{in}}\right)$$

(2)

where ${\mathrm{Q}}_{\mathrm{in}}$ means the geometric region located inside the image after segmentation, and ${\displaystyle \bigcup }_{\mathrm{k}=0}^{\mathrm{K}}$ means a union set with elements numbered from 0 to K.

$${\mathrm{S}}_{\mathrm{k}}\left({\mathrm{Q}}_{\mathrm{in}}\right)=\left({\mathrm{Q}}_{\mathrm{in}}⊝\mathrm{kB}\right)-\left({\mathrm{Q}}_{\mathrm{in}}⊝\mathrm{kB}\right)\circ \mathrm{B}$$

(3)

where B represents the structural element. Here, we used the built-in skeletonization lookup table from MATLAB, which contains a set of different structural elements, and which changes with the erosion condition. $\circ \mathrm{B}$ means an opening operation by structural element B, and $⊝ \mathrm{B}$ means an erosion operation by structural element B.

Furthermore,

$$\left({\mathrm{Q}}_{\mathrm{in}}⊝\mathrm{kB}\right)=\left(\left(\cdots \left({\mathrm{Q}}_{\mathrm{in}}⊝\mathrm{B}\right)⊝\mathrm{B}\right)\notin \cdots \right)⊝\mathrm{B})$$

(4)

represents multiple erosion, and k is the last iteration step before ${\mathrm{Q}}_{\mathrm{in}}$ is eroded to an empty set, that is:

$$\mathrm{K}=\max \left\{\left.\mathrm{k}\right|\left({\mathrm{Q}}_{\mathrm{in}}⊝\mathrm{kB}\right)\ne \varnothing \right\}.$$

(5)

The skeletonization of the image can not only maintain the topological information of the original structure but also has good resistance to slight deformation such as stretching and rotating, preserving consistency between skeletons from different imaging perspectives.

2.4. Transformation

After the skeletonization, the endpoints of the structure $\mathrm{S}\left({\mathrm{Q}}_{\mathrm{in}}\right)$ were obtained. Endpoints are extremes of arcs and correspond to tips of pattern protrusions, which have only one connected pixel in the eight neighborhoods of the skeleton. In the target, the endpoints are the tips of the contour protrusions with the largest curvature in the local arc segment, which will not change due to the change of imaging mode. After obtaining the target skeleton endpoints of the infrared image and polarization image, the endpoints were used as control points for image matching in our method. With T as the transform matrix, the transformation can be expressed as

$$\mathrm{Fixed} \mathrm{image} · \mathrm{T} = \mathrm{moved} \mathrm{image},$$

(6)

therefore, we used

$${\mathrm{T}}_{\mathrm{inv}}= \mathrm{fixed} \mathrm{points} \mathrm{matrix}/\mathrm{moved} \mathrm{points} \mathrm{matrix},$$

(7)

and then,

$$\mathrm{Registered} \mathrm{image} = \mathrm{moved} \mathrm{image}\cdot {\mathrm{T}}_{\mathrm{inv}},$$

(8)

to give the registered images.

3. Results and Discussion

The experiments were performed on both synthetic images and real-world images. The registration experiments were conducted in MATLAB2018b on a Windows10 computer with Intel Core 7th generation i7 CPU and 8 GB memory.

3.1. Synthetic Images Experiment

Two synthetic images were designed to simulate the characteristics of infrared and polarized images. One image contains rich texture details and a consistent gray value, which simulates the polarization parameter image. The other image has a large difference in gray level but few texture features, which simulates the infrared image. In the two images, the target was shifted, zoomed, and slightly rotated to simulate the different imaging positions of the two cameras. In addition, the aspect ratio of the two images was inconsistent to simulate the size differences of the different sensor chips. The registration accuracy rate of SSR reached 96.67% as calculated by IoU (Intersection of Union). From the subjective point of view, the target in the two images completely overlaps as shown in Figure 3.

We compared several feature-based algorithms such as SURF, MSER, Brisk, FAST, Harris, and MinEigen, and their checkpoint detection, and matched the pair results as shown in

Table 1. Feature-based synthetic images registration experiment.

Feature	Original Image Pair		Segmented Image Pair
	Detected Points	Matched Pairs	Detected Points	Matched Pairs
SURF	152-692	0	88-273	2
MSER	1-1107	0	0-0	0
Brisk	13-1	13	64-21	0
FAST	1-1	1	7-8	4
Harris	8-0	0	8-79	0
MinEigen	8-87540	0	8-91	0

.

By analyzing the synthetic image pair, none of the feature-based registration algorithms can achieve this registration. Therefore, only their intermediate process data discussions are listed in Table 1. Due to the huge difference in features between images, the experimental results of feature-based algorithms are not good.

In the results of the original image pair, SURF, MSER and MinEigen found a large number of feature points, but none of them were matched. FAST and Brisk found fewer mismatched feature points. Harris could not even find a feature point pair. These algorithms fail for various reasons. SURF and MSER are blob feature sensitive. SURF search found features by Hessian matrix, which means the result depends on the gradient of region, while MSER search found extremely bright or dark region; since multimodal images are nonlinear in pixel value, their gradient difference and brightness difference lead to misalignments in SURF and MSER. Brisk, FAST, MinEigen, and Harris are corner sensitive. As Table 1 shows, Brisk and Harris found corners of the geometric region in Figure 3a, but the corners in Figure 3b were submerged in texture, and no checkpoint can be paired. FAST found one pair of checkpoints, but it is insufficient for registration. MinEigen was confused by the texture and erroneously derived many points, which obstructed it from finding the best match pairs.

Briefly, this algorithm starts by segmenting the synthetic images to achieve a more consistent binary image and then carries out the registration comparison experiment with feature-based algorithms. The analysis of the segmented image pairs yielded similar results. Brisk, Harris, and MiniEigen found sufficient feature points, but could not match them. MSER could not work out feature point pairs. SURF and FAST matched a few pairs, but the results were wrong. After active contour segmentation, the angular features of the target are smooth and the original straight edges arise with slight ripples, which makes the feature registration of the segmented image poor.

The experiment shows that this kind of algorithm is extremely unsuitable in our case. Table 1 shows that feature-based registration algorithms basically find out lots of feature points from the synthetic images, but hardly match them.

The MIND [22] descriptor is tested here. However, in this challenging situation, it suffered from the large search region and did not work out the result of simulating infrared and polarization parameter synthetic image registration. The output of MIND descriptor registration is shown in Figure 4.

Thereafter, we tested the intensity-based algorithm in the synthetic infrared image and the polarized parameter image pairs.

Common intensity-based registration algorithms are monomodal. The registered results cannot be obtained by a mere monomodal intensity-based algorithm because the original synthetic image pair does not share the same gray features. Therefore, we compared the algorithms with segmented images. Some of these algorithms did not figure out the results due to their performance limitations. The following figure is the comparison result between the SSR algorithm and Mean Squares, Mattes Mutual Information Phase Correlation, Powell, and PSO.

Figure 5 shows the registration result of our SSR and above algorithms, which are not designed for multimodal image registration. Figure 5a shows the initial status of the segmented image pairs. Figure 5(b2) is an intermediate step to Figure 5(b1) in SSR; the SSR algorithm has an absolute advantage in both processing efficiency and registration accuracy in comparative experiments. None of the registration results in group c. The Mean Squares (c1) and Mattes MI (c2) algorithms support the transformation of image size, but cannot find the correct extreme values. Phase correlation (c3) achieves registration on two sides while the algorithm does not support image size transformation. The results of Powell (c4) and PSO (c5) were too misaligned. The cause of their failure varies. Mean Squares directly evaluates the pixel value of the two images. In Figure 5(c1), the targets are mostly overlayed, but the search did not find the right rotation. Mattes MI evaluates the joint histogram of two images, and this kind of method aligns images according to grayscale probability. In Figure 5(c2), the targets are overlayed in a geometrically pleasing but wrong way, which infers that the search was trapped in a local optimal. Phase correlation evaluates the phase difference of images in the frequency domain, which is a shift between two images in the spatial domain. In Figure 5(c3), the search aligned two sides for both targets and got a right rotation angle, but it stopped while one of the targets was completely overlayed. The Powell is easy to be trapped in a local optimal as well. In Figure 5(c4), Powell found the best rotation angle and stop searching. The PSO is good at global searching but poor at local searching, as we can see from Figure 5(c5), while the two targets were largely overlapped, it was difficult for PSO to converge further.

3.2. Real World Image Experiment

The infrared image was taken by Bobcat-320-Star uncooled smart InGaAs camera from Xenics, the output resolution is 320 × 256, and the spectral waveband is between 0.9–1.7 μm. The polarization parameter image was taken by polarized microarray camera TRI050S-PC from Lucid Vision and the output resolution is 1224 × 1024. The output of both cameras is in png format. We took 30 pairs of real-world images and some of them failed in all registration algorithms, as with other registration algorithms, SSR could not register a completely uniform scene. We took a fallen leaf on the lawn as an example scene to illustrate the comparative performance of each algorithm when a target appears in a uniform background. The polarization image is mainly composed of contours and textures. The veins on the leaves can be seen clearly from the image. By comparison, the infrared image is composed of gray level information, and near-infrared can image the lawn through the leaf. The data acquisition and processing flow are shown in Figure 6. This image pair can be found in https://drive.google.com/drive/folders/1zKAjsZ6le27GT9UHugWfQO6Zm2ULmY7Q?usp=share_link, accessed on 11 October 2022.

The registration accuracy rate of SSR reached 91.02% in the original image pair experiment. In contrast, SURF, MSER Brisk, FAST, Harris, and MinEigen did not work out the result due to the huge differences in features. It was unable for the nonlinear pixel value in two images to derive gradient consistent regions for SURF and extreme value consistent regions for MSER. The complex texture of the lawn in the polarized parameter image could not provide available corner information for Brisk, FAST, MinEigen, and Harris.

Infrared image and polarized parameter image share few common features. We segmented the region of interest in the image and test the feature-based algorithms again. However, the bright areas stray from the target of the infrared segmented image confusing them in getting the same features. Only SURF gave sufficient point pairs, but were mismatched. All rest algorithms tested in this experiment fail to register the segmented image pair.

Table 2. Feature-based real world images registration experiment.

Feature	Original Image Pair		Segmented Image Pair
	Detected Points	Matched Pairs	Detected Points	Matched Pairs
SURF	0-55	0	80-366	13
MSER	14-122	0	2-6	0
Brisk	0-0	0	32-552	0
FAST	0-0	0	9-259	0
Harris	17-12	0	2-30	0
MinEigen	45-40	0	2-49	1

shows the detected feature points of the algorithms, which were seldom matched either. The image segmentation helps SURF find out some match pairs, which are mismatched and helpless for achieving the correct registration.

A state-of-the-art result of MIND is given in Figure 7, but it did not work.

Compared with the registration algorithms BSO, GA, Powell, and PSO, based on the segmented image, the results are shown in the figure below, and the experimental result of manual registration is added.

It can be read from Figure 8 that intensity-based algorithms in d, e, f, and g fail to register the infrared image and polarized parameter image. Completely overlaying in a single target might prevent these algorithms from further searching. SSR is the only automatic algorithm that registers the above images.

3.3. Objective Evaluation

All the following registration results for comparisons are given by the intensity-based registration algorithms. The running time and Intersection of Union (IoU) are used as indicators for comparison. Intensity-based algorithms could not register the original image directly. The time indicators are calculated from the running duration experiment of the segmented image. The average time for image segmentation was 10.73 s on our device, which was excluded from the comparison.

The accuracy rate of registration was calculated by the IoU formula as follows:

$$\mathrm{IoU}=\mathrm{Area} \mathrm{of} \mathrm{Intersetion}/\mathrm{Area} \mathrm{of} \mathrm{Union} = (A_{fixed}\cap A_{moved})/(A_{fixed}\cup A_{moved})$$

(9)

ranging from 0 to 1. $A_{fixed}$ and $A_{moved}$ indicates the region of targets in the polarized image and infrared image. When the targets in the two images are completely overlapped, this means the two images are fully registered, the IoU is 1.

Since the multi-modal operator MIND registration result cannot be used for IoU calculation, it is omitted here. Additionally, it takes 154.41 s to register the synthetic image and 16.17 s to register the real-world image. It can be seen from

Table 3. Indices of intensity-based registration experiment.

Algorithm	Indices of Synthetic Image		Indices of Real World Image
	Time	IoU	Time	IoU
SSR	1.31 s	96.67%	1.76 s	91.02%
Manual	9 min+	93.49%	15 min+	94.49%
Mean Squares	5.60 s	73.35%	/	/
Mattes MI	4.20 s	82.52%	/	/
Phase correlation	0.85 s	69.84%	/	/
Powell	10.36 s	69.29%	6.99 s	53.94%
PSO	10.72 s	68.63%	6.47 s	56.08%
BSO	/	/	4.95 s	53.99%
GA	/	/	9.19 s	55.80%
MIND	154.41 s	/	16.17 s	/

that the performance of the SSR algorithm is the best in the infrared and polarization parameter registration experiments, except for manual registration in the real-world experiment, which took more than 15 min to process.

For binary images with obvious consistent gray areas, registration methods based on intensity can be tested, such as BSO and GA. Those methods are mostly time-consuming, and lack robustness for distortion and field curvature in images taken by different cameras. In contrast, our SSR can be endowed with high operation speed and robustness to optical distortion by integrating skeleton endpoints as the control points for image transformation.

A total of 30 pairs of real-world images were registered by SSR. The IoU results were shown in Figure 9. It can be seen that SSR performed good registration in all the pictures. Since the IoU is affected by the segmentation results, the larger deviation in one of the scenes is attributed to the impure segmentation. In our practical applications, SSR performed well enough to meet the registration needs of polarization parameters and infrared images.

4. Conclusions

For the registration of infrared images and polarization parameter images, we completed the two following works: one to obtain the common registration basis for the image to be registered by segmentation; the other to obtain the registration control points by skeleton to improve the robustness to viewing angle difference and image deformation. The experimental results show that the performance of SSR is better than that of two classical registration algorithms and state-of-art multimodal algorithm in infrared and polarization parameter image registration. Objectively, compared with the traditional algorithms that can run the results, the average accuracy and operation speeds are increased by 38.36% and 150.56%, respectively.

From the results of image fusion in Figure 6 after registration, we can see that the vein information from the visible polarization parameter image, and the perspective information from the infrared image, are all reflected. The amount of information in subsequent fusion images increased. It also proved the validity and application value of the SSR registration algorithm. The future performance of SSR in optical multi-sensor detection is expected.

In future steps, we can consider using the classification method of artificial intelligence to segment the same object of the two images accurately, and then carry out the registration based on this.