3.2.2. Registration Results of CT and MR Images

To appreciate registration performance of all evaluated methods operating on CT and MR images from NA-MIC database, these methods are implemented to correct the synthetic deformation that is applied to the float image, where the CT and MR images will be used as the reference and float images, respectively. Here, the liver CT and MR images of size 256 × 256 × 32 are intra-operatively and pre-operatively acquired, respectively. Due to strong differences in image contrast between CT and MR images, their registration is difficult.

Table 3 lists the TRE and *P* values of *t*-test for the FMIND method and other methods. As you can see, among all the compared methods, the FMIND method has the highest registration accuracy by providing the lower TRE than other methods. Meanwhile, all the *P* values are less than 0.003, which indicates the significant difference between the FMIND method and any other method in terms of TRE.

**Table 3.** The TRE for all evaluated methods and the *P* values for the *t*-test between the FMIND method and other compared methods operating on the three-dimensional CT-magnetic resonance (3D CT–MR) image pairs.


Figure 10 shows the registration results of 3D CT-MR images for all the evaluated methods. As shown in Figure 10c,d, the ESSD and MIND method cannot effectively correct the deformation that is involved in the MR image. The FMIND method can produce a more similar registration result to the reference image that is shown in Figure 10a than the ESSD and MIND methods. When compared with the most competitive HLCSO method, the proposed method performs better in that it can correct the deformation of some tissues more effectively, as indicated by the three red boxes that are shown in Figure 10e,f.

**Figure 10.** The registration results of all evaluated methods operating on 3D CT–MR images. (**a**) CT image (reference image); (**b**) MR image (float image); (**c**) ESSD; (**d**) MIND; (**e**) HLCSO; and, (**f**) FMIND.

Table 4 lists the implementation time of all the evaluated methods. The comparison indicates the advantage of the FMIND method in computational efficiency. Here, it should be noted that the implementation time for all evaluated registration methods in Table 4 is very similar to that in Table 2, because the used CT and MR images have the same size (256 × 256 × 32) to T1, T2, and PD images.


**Table 4.** Computation time for all evaluated methods operating on the 3D CT–MR image pairs.

### 3.2.3. Registration Results of CT and PET Images

The 3D whole body CT-PET images from NA-MIC database are also used to demonstrate the advantage of the FMIND method. Here, the CT and PET images of size 168 × 168 × 149 are the reference image and the float image, respectively. It is difficult to realize accurate registration of CT images and blurry PET images of low resolution.

Figure 11 shows the registration results of the 3D CT-PET images for the ESSD, MIND and HLCSO, and FMIND methods. It can be observed that the ESSD and MIND methods cannot correct the deformation in the regions that are marked with the red boxes in Figure 11c,d well. By comparison, the registration results of the HLCSO and FMIND methods are more similar to the reference image shown in Figure 11a than the ESSD and MIND methods.

**Figure 11.** Registration results of all evaluated methods operating on the 3D CT–PET images. (**a**) CT (reference image); (**b**) PET (float image); (**c**) ESSD; (**d**) MIND; (**e**) HLCSO; and, (**f**) FMIND.

Table 5 lists the mean and standard deviation of TRE for all evaluated methods operating on 3D CT and PET images. The comparison of TRE values shows that the HLCSO method provides the minimum mean (2.6 voxel) and standard deviation (0.7 voxel) of TRE among all the compared methods. However, the mean (2.8 voxel) and standard deviation (0.9 voxel) of TRE for the FMIND method are lower than those for the ESSD and MIND methods. The reason can be explained in this way. For the PET image, its contrast and resolution are poor and the edge features are not obvious. Therefore, the FMIND method is slightly inferior to the HLCSO method in the registration of 3D CT-PET images. However, the proposed FMIND method can still provide better structural representation results than the ESSD and MIND methods, thereby leading to its improved registration accuracy than the latter.


**Table 5.** TRE for all evaluated methods operating on the 3D CT–PET image pairs.

Table 6 lists the calculation time of the various methods operating on CT and PET images. It can be seen from Table 6 that, as compared with other methods, the calculation time of the FMIND method is significantly reduced because the spatial constraint based on the FMIND, to a certain extent, helps to reduce the number of variables that are required by the deformation model. Especially, when compared with the HLCSO method, although the FMIND method has slightly lower registration accuracy, its computational efficiency is more than two times higher. Besides, as compared with the calculation time listed in Table 2, more calculation time will be involved in the registration of CT-PET images because their size (168 × 168 × 149) is bigger than that of T1, T2, and PD images.

**Table 6.** Computation time for all evaluated methods operating on the 3D CT–PET image pairs.


### 3.2.4. Registration Results of Real MR Images

The MR images from RIRE database are chosen to verify the superiority of the FMIND method in registering the real MR images, where the T1 and PD weighted MR images are used as the reference and float images, respectively. These MR images were acquired while using a Siemens SP Tesla scanner, among which the T1 and PD image volumes were obtained with an echo time of 15 ms and 20 ms, respectively [42].

Here, we will only compare the proposed method with the MIND and SSC methods, which are most similar to our method. Table 7 lists the TRE values of the three methods. Clearly, the SSC method generally provides slightly smaller TRE values than the MIND methods. The two methods are outperformed by the FMIND method in terms of registration accuracy. The comparison of TRE values indeed demonstrates the effectiveness and advantage of the FMIND method in correcting the deformation of real MR images.

**Table 7.** The TRE for the MIND, self-similarity context (SSC), and FMIND methods operating on the real T1-PD image pairs.


### 3.2.5. Registration Results of Real MR and US Images

We will use the pre-operative T1 weighted MR and intra-operative post-resection US images of 13 patients [43] from BITE database for registration performance appreciation to further demonstrate the practicality of the FMIND method. In [43], the MR images were obtained a few days before the surgery while the post-resection 2D US images were acquired while using Philips HDI 5000 ultrasound machine with a P7-4 MHz phased array transducer and they were reconstructed into ultrasound volume with a voxel size of 1 mm. The used MR data contain the tumor, which is replaced by the resection cavity, and thus will not exist in the post-resection US images. Therefore, to register 3D MR to 3D US images is highly challenging. For each patient, 15 landmarks in average selected in [43] are used for TRE evaluation.

Table 8 lists the TRE values of the MIND, SSC, and FMIND methods. Clearly, the SSC method provides smaller TRE values than the MIND method. The FMIND method also performs better than the MIND method, in that the introduction of foveated nonlocal self-similarity ensures more effective structural representations of MR and US images. Please note that the proposed method cannot significantly outperform the SSC method for registration of US-MR images due to the disadvantageous influence of speckle noise that is inherent in US images.

**Table 8.** The TRE for the MIND, SSC, and FMIND methods operating on the real US-MR images of 13 patients.


### **4. Conclusions**

In this paper, we have proposed a novel non-rigid multi-modal 3D medical image registration method that is based on the foveated independent neighborhood descriptor. The advantages of the proposed method lie in two aspects. Firstly, the proposed FMIND can effectively capture the structural feature information of 3D medical images, thereby providing better structural representations than the existing approaches. Secondly, the FMIND based spatial constraint method can help to reduce the number of non-rigid transformation parameters because the FMIND contains the corresponding relationship of the local spatial information at the same pixel in the reference and float images, thereby providing an effective means for solving the high-dimensional optimization problem that is involved in the medical image registration. Experiments on 3D ultrasound, CT, PET, T1, T2, and PD weighted MR images demonstrate that our method can provide higher computational efficiency and higher registration accuracy as compared with the HLCSO, ESSD, MIND and SSC methods, except that its TRE is slightly bigger than that of the HLCSO for CT-PET image registration. Future work will be focused on the acceleration of the method without compromising registration accuracy by using sparse data sampling and parallel data processing strategies to facilitate its clinical applications.

**Author Contributions:** F.Y. performed the experiments, analyzed the data and drafted the manuscript. X.Z. and M.D. supervised the research and contributed to the article's revision.

**Funding:** This work was partly supported by the National Natural Science Foundation of China (NSFC) (Grant No.: 61871440, 61861004), and the National Key Research and Development Program of China (Grant No.: 2017YFB1303100).

**Conflicts of Interest:** The authors declare no conflict of interest.
