Experimental Analysis of Feature-Based Image Registration Methods in Combination with Different Outlier Rejection Algorithms for Histopathological Images ^†

Abstract

Registration involves aligning two or more images by transforming one image into the coordinate system of another. Registration of histopathological slide images is a critical step in many image analysis applications including disease detection, classification, and prognosis. It is very useful in Computer-Aided Diagnosis (CAD) and allows automatic analysis of tissue images, enabling more accurate detection and prognosis than manual analysis. Due to the complexity and heterogeneity of histopathological images, registration is challenging and requires the careful consideration of various factors, such as tissue deformation, staining variation, and image noise. There are different types of registration and this work focuses on feature-based image registration specifically. A qualitative analysis of different feature detection and description methods combined with different outlier rejection methods is conducted. The four feature detection and description methods experimentally analyzed are Oriented FAST and rotated BRIEF (ORB), Binary Robust Invariant Scalable Key points (BRISK), KAZE, and Accelerated KAZE, and the three outlier rejection methods examined are Random Sample Consensus (RANSAC), Graph cut RANSAC (GC-RANSAC), and Marginalizing Sample Consensus (MAGSAC++). The results are visually and quantitively analyzed to select the method that gives the most accurate and robust registration of the histopathological dataset at hand. Several evaluation metrics, the number of key points detected, and a number of inliers are used as parameters for evaluating the performance of different feature detection–description methods and outlier rejection algorithm pairs. Among all the combinations of methods analyzed, BRISK paired with MAGSAC++ generates the most optimal registration results.

1. Introduction

Histopathology is the examination of cells and tissues under a microscope to detect diseases and characterize them. Digital pathology generates high-definition histopathology sections available for automatic analysis, enabling computer-aided diagnosis (CAD), disease grading, prognosis, and other image processing applications. Image registration is the process of putting two images in the same frame of reference by spatially changing one image to another [1,2,3,4,5]. It is a vital step in the pre-processing of images where information from multiple images must be combined [6,7,8]. Feature-based registration (FBR) is a type of registration that detects key points and aligns images by aligning the corresponding key points between them. FBR detects regions of interest based on distinctiveness, repeatability, and high information, giving better correspondence matches and fast and accurate registration. Once the key points are extracted, the next step is to match the corresponding points of the image pair. Not all the points are good matches (inliers). Removing the bad matches (outliers) from the set of all matched key points can significantly improve the quality of registration. This is performed using an outlier rejection or removal algorithm, which refines the registration to produce a set of inliers. The main objective of this work is to focus on methods that combine feature detection and description (FDD) algorithms. Here, four FFD methods, namely Oriented FAST and rotated BRIEF (ORB), Binary Robust Invariant Scalable Key points (BRISK), KAZE, and Accelerated KAZE (AKAZE) are combined with three outlier removal algorithms, RANSAC (Random Sample Consensus), GC-RANSAC (Graph Cut RANSAC), and Marginalizing Sample Consensus (MAGSAC++) and their results are evaluated. The combination of each feature detection algorithm with the outlier rejection algorithms is shown in Figure 1. Several evaluation metrics, also called the similarity metric, are used for quantitative analysis of the registration results produced by different methods.

Each of the four FDD methods is combined with three outlier removal methods, resulting in a total of 12 combinations. They are ORB with RANSAC (OR), ORB with GC-RANSAC (OGR), ORB with MAGSAC (OUM), BRISK with RANSAC (BR), BRISK with GC-RANSAC (BGR), BRISK with MAGSAC (BM), KAZE with RANSAC (KR), KAZE with GC-RANSAC (KGR), KAZE with MAGSAC (KUM), and AKAZE. A visual and quantitative analysis of each FDD method with different outlier rejection methods is carried out to find the method that performs best for the histopathological dataset at hand.

2. Related Work

Scale-Invariant Feature Transform (SIFT) is among the earliest developed FDD algorithms and has since become the most popular method for feature detection for all kinds of applications. SIFT revolutionized FDD as it converts image data into scale-invariant coordinates. These make the features invariant to scale and rotation [9,10] and partially invariant to illumination [11,12]. Most of the other handcrafted FDD methods are based on SIFT and developed as an improved version of it. Speeded-Up Robust Features (SURF) is a modified accelerated version of SIFT [13]. It has all the properties of it and is robust to changes in illumination, rotation, and scale but is much faster [13,14]. ORB is available as an open-source library and free to use [15,16], making it a popular alternative to the patented SIFT and SURF. It has a comparable matching performance to SIFT and SURF and its speed makes it applicable for real-time performance [17]. BRISK combines the FAST detector with a binary string descriptor by combining the results of simple brightness tests [15,16]. It has a high-quality performance similar to SIFT and SURF and is several orders of magnitude faster than SURF. KAZE is a scale-invariant non-linear scale space detector [18]. As a non-linear variant of SIFT, it has a similar good-quality performance and high computational cost [10]. Thus, an accelerated version of KAZE was designed called A-KAZE. The method used to create the non-linear multiscale detector is several orders of magnitude faster than KAZE.

Before computing the transformation, the presence of incorrect corresponding points that negatively impact the accuracy of the estimated transformation must be identified and removed [19]. This bad match removal process is conducted using an outlier rejection algorithm. RANSAC is simple yet powerful and hence the most widely used outlier rejection algorithm [20,21,22]. It is an iterative process that produces highly accurate estimations of the transformation parameters [22]. RANSAC, though robust, has its limitations that affect its precision and efficiency [22]. Several variations with efforts to improve its efficiency have been introduced [20]. Before choosing any method, the properties of all methods must be carefully considered as designing a single universal approach that is applicable under all scenarios is unrealistic due to the heterogeneity of the image registration process [6,8].

3. Materials and Methodology

The first step in registering images is detecting and describing the key points that are later matched to align the images. A detector is a feature-detection algorithm used to extract distinctive regions from images [13]. A descriptor is an algorithm that encodes the information of the extracted region like its pixel coordinates, orientation, and luminance that can uniquely identify it. Next, the extracted key points must be filtered to remove the outliers that negatively affect the accuracy of the registration. Once the outliers are removed, the corresponding points between the image pair are matched to estimate the transformation matrix. Based on the transformation matrix, the key points of the moving image are warped to align with the base image and registration is complete.

The analysis of the various methods mentioned has been carried out using histopathology images taken from GlaS@MICCAI’2015: Gland Segmentation [23]. It is a publicly available dataset for the accurate segmentation of glands in colon histology images. The images have been augmented using ‘torchvisions. transforms’ module of the PyTorch library. The ‘Random Perspective’ transform applies a random perspective transformation to the images. It is useful to simulate the effect of viewing an object from different angles. The ‘distortion_scale’ parameter controls the amount of distortion applied to the image. A larger value will result in a more distorted image. Here, the scale is set to 0.6. The ‘p’ parameter controls the probability that the transformation will be applied to an image. The value of p is set to 1.0 and the transform will always be applied. Figure 2 shows a sample image and a set of augmented images.

A flowchart of the methodology is shown in Figure 3. Image 1 is the first image in the dataset and is used as the fixed or reference image. Image 2 is the moving image and it changes with each iteration. ‘n’ is the total number of images being registered and hence the number of times the loop is executed.

The general steps involved in the image registration are as follows:

• Step 1: Input image pair for registration (image1 and image2).
• Step 2: Initialize the feature detector.
• Step 3: Generate key points for image1 and 2 as kps1 and kps2, respectively.
• Step 4: Generate descriptors for kps1 and kps2 as des1 and des2, respectively.
• Step 5: Match key points using Brute Force Hammer.
• Step 6: Extract the matched key points.
• Step 7: Filter the matched key points using the outlier rejection method.
• Step 8: Find the transformation matrix that maps image2 to image1.
• Step 9: Find the height, width, and channel of image1.
• Step 10: Warp image2 using the shape of image1 and the transformation matrix.

All steps in the process except Step 2 and Step 7 are the same. The instance of a particular FDD to be examined is created in Step 2. The outlier rejection method to be analyzed is used in Step 7. The matcher used for all the methods is Brute Force Hammer. It is based on a brute force technique in which all possible matches between two sets of features are explored and compared using the Hamming distance metric. The Hamming distance is a measure of the difference between two binary strings, with the distance determined as the number of bit positions that differ between the two strings. Using the Hamming distance, all feature descriptors in one image are compared to all feature descriptors in the other image. The best match between the two features is the one with the shortest Hamming distance.

Similarity Metrics

The results obtained from different methods of image registration techniques are evaluated using 7 similarity measures or evaluation metrics. These metrics provide a quantitative analysis of visual and semantic similarity between the input and the registered image. The 7 evaluation metrics are as follows:

• Structural Similarity Index (SSIM): The structural information in two images is measured using the SSIM, which is a comprehensive image quality assessment metric to determine how similar the two images are. It accounts for the disparities in luminance, contrast, and structural composition between the two images.
• Mean Squared Error (MSE): This calculates the average of the squared differences between the pixel values of the two images.
• Root Mean Squared Error (RMSE): This is the square root of the MSE. It measures the standard deviation of the differences between the corresponding pixels of the two images.
• Peak Signal to Noise Ratio (PSNR): PSNR computes the ratio of maximum pixel value possible (255 for a grey image) to the MSE between the two images.
• Spatial correlation coefficient (SCC): This calculates the correlation between the registered and reference images’ pixel intensities. It is computed as the ratio of the covariance of the two images to the project of their standard deviation.
• Universal Quality Index (UQI): Like SSIM, this also considers luminance, contrast, and structural similarity between any image pair. But UQI is based on statistical similarity whereas SSIM is based on structural similarity between images. UQI takes both local and global features into account and ranges from 0 to 1. A UQI of 1 indicates identical images and 0 indicates no similarity.
• Visual Information Fidelity (VIF): This is a similarity metric that assumes that the output image after registration must presume the visual and structural similarity of the original image. It compares the contrast and illuminance of each corresponding pixel of the images being compared. VIF values range from 0 to 1. The higher the value, the greater the similarity, and the lower the value, the less similar the input and the output images are.

4. Results

Figure 4 shows the final output of registration after bad matches are removed and the transformation matrix is applied. Visibly, KAZE and BRISK produced good and comparable results and some slightly deformed images were produced by AKAZE.

Precision is calculated as the ratio of inliers to the total number of extracted key points. As shown in

Table 1. Precision based on total number of matches and inliers.

Methods	Total Matches	Inliers	Precision
OR	158	84	0.5325
OGR	158	92	0.5840
OUM	158	92	0.5864
AR	130	73	0.5638
AGR	130	78	0.6003
AUM	130	78	0.5987
KR	242	145	0.6000
KGR	242	159	0.6580
KUM	242	159	0.6585
BR	403	269	0.6663
BGR	403	281	0.6961
BUM	403	281	0.6986

, RANSAC produces the lowest number of inliers and produces registration with the least precision when paired with any of the FDDs. The outlier rejection ratio of GC-RANSAC and MAGSAC++ is very similar and thus has a very similar precision rate. ORB and AKAZE produce the lowest number of feature points. This means they are fast but not very robust to illumination, color, contrast, or intensity variation between images. KAZE generates more feature points and produces better alignment than those two methods but is much slower. BRISK produces a high number of key points, and its speed is comparable to ORB and AKAZE. If we observe precision values just based on the number of key points generated and the number of good matches among them, BRISK and MAGSAC++ produce the most accurate registration.

Table 2. Evaluation metrics of different feature-based image registration methods.

Methods	SSIM	MSE	RMSE	PSNR	SSC	VIF	UQI
OR	0.8529	1305.1034	25.6543	25.6542	0.2156	0.3389	0.8457
OGR	0.8762	1278.2864	23.0218	22.7963	0.2554	0.3704	0.8514
OUM	0.8904	822.9807	20.9947	23.1479	0.2659	0.3776	0.8557
AR	0.8839	830.5151	21.5185	23.2056	0.2708	0.3835	0.8557
AGR	0.8863	605.3339	20.3041	23.2952	0.2694	0.3823	0.8623
AUM	0.8993	754.2623	19.3764	24.0158	0.2911	0.4039	0.8599
KR	0.9012	356.8790	17.9529	23.4708	0.2634	0.3821	0.8667
KGR	0.9206	265.5308	15.6120	24.6149	0.3011	0.4145	0.8714
KUM	0.9255	239.9796	14.9799	24.8995	0.3090	0.4216	0.8730
BR	0.9375	190.4855	13.0487	26.1484	0.3432	0.4597	0.8771
BGR	0.9431	163.8165	12.0929	26.7919	0.3580	0.4764	0.8786
BUM	0.9485	134.1033	11.3972	27.1108	0.3656	0.4844	0.8799

, given below, shows the values of the averages of different evaluation metrics calculated after registering 100 images together. The values were consistent while registering lower as well as greater numbers of images.

5. Discussion

After experimentation and quantitative analysis of the results, it is clear that ORB has the least optimal performance in comparison to the other approaches examined in this paper. It has the lowest similarity indices, both local and global, low spatial correlation, and low visual information fidelity. The similarity measures and number of features detected by ORB and AKAZE are comparable but the latter has a notably lower rate of mean squared errors and much higher visual accuracy. KAZE and BRISK perform significantly better than the other two methods in almost all areas and they have uniform structural and statistical indices. BRISK in combination with any one of the three bad match removal techniques gives the optimal result. Its spatial correspondence, similarity indices, PSNR, and visual accuracy information rates are much greater and the sum of squared errors is minimal between the original and registered images.

Amongst the three algorithms for outlier rejection, RANSAC gives the least satisfactory results. The difference in the error rates, signal-to-noise ratio, and SSIM and UQI values produced by registration using RANSAC and the other two methods are significant. GC-RANSAC and MAGSAC++ give much more accurate results and their similarity measures are comparable. MAGSAC++, however, outperforms the other two techniques; the similarity between the input and output image is the highest, and the error rates are minimal. The spatial correspondence, PSNR, and VIF of registered images obtained using this algorithm are the highest. Figure 5 and Figure 6 below depict the different evaluation metric values from Table 2 graphically for a better understanding of the tabular data. The X-axis represents the different combinations of feature detection and outlier removal methods, while the Y-axis shows the values of the similarity measures. From the experimental data of the various similarity measures, precision and runtime values, and their quantitative analysis, it becomes clear that for the most accurate and robust feature-based registration of histopathological images with minimal computational cost, BRISK with MAGSAC++ is the best choice. Figure 5 shows a graph representing SSIM, SSC, UQI, and VIF values for different methods. Figure 6 and Figure 7 represent graphs for PSNR and RSME values calculated for different combinations of methods.

6. Conclusions

After thorough analysis of the performance of all the methods using various similarity measures, we can say that traditional RANSAC in comparison with others gives the least optimal result when used with any of the four FDDs. MAGSAC++ significantly outperforms other outlier removal techniques when combined with any of the FDDs. The mean squared error for ORB was the highest and it is the only method to give visibly faulty and incomprehensible output images. ORB and AKAZE give comparable outputs that show low similarity between input and output images and a low number of key points as compared to the other two methods. KAZE also yields precise results but is much slower than the rest. BRISK produces the best registration between images. It produces a maximum number of corresponding points and is still fast. Thus, BRISK combined with MAGSAC gives the best registration result. High accuracy and reliability are a must while processing medical images and different applications have different requirements, thus the method of choice for registration or any image processing techniques will also vary greatly. These results are specific to histopathology images and can be different depending on the imaging modality or even between histology images with different properties.