Reverse-Net: Few-Shot Learning with Reverse Teaching for Deformable Medical Image Registration

Abstract

Multimodal medical image registration has an important role in monitoring tumor growth, radiotherapy, and disease diagnosis. Deep-learning-based methods have made great progress in the past few years. However, its success depends on large training datasets, and the performance of the model decreases due to overfitting and poor generalization when only limited data are available. In this paper, a multimodal medical image registration framework based on few-shot learning is proposed, named reverse-net, which can improve the accuracy and generalization ability of the network by using a few segmentation labels. Firstly, we used the border enhancement network to enhance the ROI (region of interest) boundaries of T1 images to provide high-quality data for the subsequent pixel alignment stage. Secondly, through a coarse registration network, the T1 image and T2 image were roughly aligned. Then, the pixel alignment network generated more smooth deformation fields. Finally, the reverse teaching network used the warped T1 segmentation labels and warped images generated by the deformation field to teach the border enhancement network more structural knowledge. The performance and generalizability of our model have been evaluated on publicly available brain datasets including the MRBrainS13DataNii-Pro, SRI24, CIT168, and OASIS datasets. Compared with VoxelMorph, the reverse-net obtained performance improvements of 4.36% in DSC on the publicly available MRBrainS13DataNii-Pro dataset. On the unseen dataset OASIS, the reverse-net obtained performance improvements of 4.2% in DSC compared with VoxelMorph, which shows that the model can obtain better generalizability. The promising performance on dataset CIT168 indicates that the model is practicable.

1. Introduction

Medical image registration technology is the basis for accurate image fusion. Medical imaging techniques can be broadly divided into two categories: structural imaging and functional imaging. Structural imaging generally refers to X-rays, computed tomography (CT), and so on, which have high spatial resolution and can reflect the anatomical structure of tissues, while functional imaging generally refers to positron emission computed tomography (PET), single-photon emission computed tomography (SPECT), and so on, which have poor spatial resolution but can record the information of tissue metabolism and neural activity. In the process of diagnosis, it is often necessary to fuse the medical image information of different modalities to improve the accuracy of clinical diagnosis. However, due to different imaging devices, different acquisition moments, and the different acquisition perspectives of medical images of different modalities, the spatial location of images of the same object in different modalities is shifted. To obtain effective fusion information, it is necessary to use image registration techniques to align the images of different modalities.

Deformable image registration can be divided into traditional methods and deep learning-based methods. Although conventional methods such as SyN [1], Demons [2], and optical flow [3] methods have achieved satisfactory performance after continuous parameter tuning, the higher time cost is not suitable for clinical treatment. Compared with traditional methods, deep-learning-based methods can provide fast registration [4] and improve registration accuracy through powerful feature extraction and learning capabilities.

Deep-learning-based methods can be divided into supervised methods and unsupervised methods. Supervised registration is achieved by minimizing the difference between the generated deformation field and the ground-truth using elaborate ground-truth. Salehi et al. [5] used mean square error to train regression neural networks for slice-to-volume registration and volume-to-volume registration scenarios. The model was applied to the registration between the T1 images of an infant brain and the T2 images of a fetus brain. Yan et al. [6] proposed an adversarial image registration model, in which the generator directly estimated the transformation parameter, and the discriminator determined whether the image pair was aligned. By introducing Euclidean distance and adversarial loss to construct the loss function, the model achieved better performance on the MR–TRUS dataset. Recent advances in using organ anatomy labels facilitate medical image registration. He et al. [7] proposed a model that utilizes the complementary nature of the joint model of medical image registration and segmentation to bring improvements in the complex scenes and few-shot situation. The diversity of the enhanced data is maintained by embedding a perturbation factor in the alignment to increase the activity of the deformation. The model finally achieved excellent results. Zhou et al. [8] provided a high-quality segmentation label for subsequent MR–CBCT multimodal registration using APA2Seg-Net. Hering et al. [9] proposed to align the surrounding structure by combining the complementary advantages of segmentation labels and local distance metrics. Compared with label-driven deep learning frameworks, its Dice scores were significantly improved. The above models achieve better performance, but the performance of the registration depends on the quality of the labels. Since segmentation labels are labeled by experienced medical experts, it is necessary to develop an unsupervised registration framework that does not require labels.

Unsupervised registration is achieved by reducing the difference between fixed and warped images. Tang et al. introduced adaptive convolution and adaptive transformer to enhance the ability of global semantic extraction. Compared with other fusion methods, satisfactory fusion results have been achieved [10]. In the field of multimodal registration, many methods are based on similarity metric training. Han et al. [11] proposed an unsupervised dual-channel registration network to resolve the deformation between preoperative MR and intraoperative CT. The model achieved excellent results in MR–CT registration. Since the above registration methods ignore the inverse consistency of the transform between a pair of images and the deformation field is only required to be locally smooth, the overlap of the deformation field cannot be completely avoided. Therefore, Zhang et al. [12] proposed to avoid folding in the transform by combining the traditional smoothness constraint with the anti-folding constraint. Mahapatra et al. [13] used GAN networks for multimodal retinal and cardiac image registration; the model outperformed the reverse deep learning-based B spline method by combining conditional and cyclic constraints [14]. For multimodal registration, it is difficult to correlate two modal images because the intensity of each modal image is different. Sun et al. [15] used image intensities and gradients to align each image pair for real-time registration. The above unsupervised registration methods do not require ground-truth and have attracted a lot of attention from researchers in recent years. However, unsupervised registration is easily misaligned on blurry structures and warped on task-independent backgrounds. Therefore, it is crucial to develop few-shot learning.

Few-shot learning trains a model from limited labeled data and reduces the need for data [16]. In medical image analysis, few-shot learning is urgently needed due to the lack of medical image data, and it has been successful in many scenarios. For example, He et al. [17] proposed few-shot learning deformable medical image registration framework. It achieved satisfactory results on three datasets using only five segmented labeled data. In addition, few-shot learning has been successful in medical image segmentation [18], object detection [19], and histopathology image classification [20].

Inspired by the above methods, we propose a framework for deformable image registration based on few-shot learning. By redesigning the network structure, our model achieves satisfactory registration performance. The main work is as follows:

• We present a multimodal few-shot learning framework for deformable medical image registration. The border enhancement network is used to enhance the ROI boundaries of T1 images and provide quality data for subsequent pixel alignment network. The coarse registration network roughly aligns the T1 image with the T2 image, which can reduce image discrepancies. The pixel alignment network generates more smooth deformation field. The reverse teaching network teaches border enhancement network rich structural knowledge. By combining the four networks, the model can achieve better registration performance by using few segmentation labels.
• We introduce reverse teaching which can improve the performance of the border enhancement network with only few segmentation labels. It utilizes the deformation field generated by the pixel alignment network to generate additional training data (warped images and warped segmentation label pairs), thus it reversely teaches the border enhancement network more structural knowledge. Therefore, the border enhancement network with better performance will provide clearer image of the ROI boundary for the pixel alignment network, finally improving the registration performance.
• The model is trained on the IXI dataset and evaluated on the MRbrain (MRBrainS13DataNii-Pro), SRI24, CIT168, and OASIS datasets. The experimental results show that the model has better performance and generalization ability.

2. Related Work

2.1. Deep-Learning-Based Unimodal Medical Image Registration

Many researchers have introduced deep learning into the field of medical image registration and have proposed many models based on deep learning. Initially, in the deep iterative approach, deep learning is used to learn the similarity metric of two images. As deep learning continues to develop in the field of medical image registration, the unsupervised registration framework with the spatial transformation network (STN) [21] after the convolutional neural network was proposed by de Vos et al. [14] first. Subsequently, a large amount of STN-based unsupervised deformable image registration frameworks emerged. For example, Balakrishnan et al. [22] proposed VoxelMorph, which uses U-net [23] as the network backbone for unsupervised brain MR image registration. In a further development, Yang et al. [24] introduced the transformer to the field of medical image registration, enabling the model to acquire local information while obtaining rich global information. The model eventually achieved good results. Many transformer-based models have subsequently emerged, such as the better-known TransMorph [25]. In the same year, Yang et al. [26] introduced graph convolution to medical image registration again. The model aims to capture high-quality long-range dependencies and expand the receptive field. However, the above models are focused on unimodal registration and still need to be explored in the field of multimodal registration.

2.2. Deep-Learning-Based Multimodal Medical Image Registration

Compared to unimodal registration, multimodal registration can provide physicians with more comprehensive information. Since most similarity metrics are focused on the field of unimodal registration, there is a lack of research in the field of multimodal registration [27,28]. Therefore, Sloan et al. [29] proposed to incorporate the inverse consistency of the learned spatial transformations into the training network to add additional constraints. The model achieved better performance in T1 and T2 brain image registration compared with the MI-optimization-based image registration methods. Liu et al. [30] proposed a modally independent neighborhood descriptor of multi-dimensional tensor (tMIND) to measure the similarity between two images. The model outperformed MI-based [31] and MIND-based [32] multimodal registration methods. Chen et al. proposed a biogeography-based optimization algorithm with elite learning (BBO-EL). Firstly, the search space of the BBO algorithm is enhanced by proposing a hybrid full migration operator. Secondly, the accuracy and convergence speed of multimodal medical registration are greatly improved by improving the low bound and the up bound of the whole population’s quality through some operations [33]. However, although the above methods propose corresponding solutions for multimodal registration, the similarity metric is still not good enough to evaluate the similarity between two images [34]. In order to use the well-developed unimodal registration similarity metric, Cao et al. [35] proposed to use an image synthesis method to synthesize pseudo-CT images and pseudo-MR images with the same anatomical structure. In this way, they converted the multimodal image registration task into a unimodal image registration task. However, although this method uses a unimodal registration similarity metric, the performance of the model is affected by the quality of the synthesized images [36].

3. Materials and Methods

The multimodal medical image registration framework based on few-shot learning is shown in Figure 1. It consists of boundary a enhancement network, a pixel alignment network, a coarse registration network, and a reverse teaching network. The boundary enhancement network is used to enhance the ROI boundaries of T1 images to provide high-quality data for the subsequent pixel alignment network and reduce registration distortion and misalignment caused by blurred ROI boundaries. The coarse registration network roughly aligns the T1 image with the T2 image, which can reduce the image discrepancy, effectively reduce the iteration time for subsequent neural network model training, and improve the efficiency of model training. The pixel alignment network generates more smooth deformation fields. Reverse teaching aligns the warped, segmented, labeled image with the warped image and maps the supervisory information in the warped, segmented, labeled image to the unlabeled image. Through the above method, rich structural knowledge is transmitted for the border enhancement network in reverse, which improves the performance of the border enhancement network. With the four networks mentioned above, we achieve deformable medical image registration with only a few segmentation labels. The detailed process of training is described in the subsequent sections.

3.1. Border Enhancement Network

The border enhancement network enhances the ROI boundaries of T1 images to provide quality data for the subsequent pixel alignment network, as shown in Figure 2. Let $x\in \mathcal{R}$ represent the sample count in the dataset, and $S\supset \left\{s^1,s^2,s^3\dots \dots s^x\right\}$ and $T\supset \left\{t^1,t^2,t^3\dots \dots t^x\right\}$ represent moving images and fixed images, respectively.

We achieve the T1 image ROI border enhancement by the border enhancement network. When the ROI border is blurred and missing, we fill the zero to the corresponding image border by the border enhancement network [37]. The border enhancement network consists of five downsampling blocks and two fully connected layers. The detailed structure is shown in Figure 3. In this process, we need to find the correspondence between T1 segmentation labels (as shown in Figure 4) and T1 images. The whole transformation process is as follows:

$$\left(\begin{array}{c}{x}_i^s\\ y_i^s\end{array}\right)=\left[\begin{array}{ccc}{\theta }_{11}& {\theta }_{12}& {\theta }_{13}\\ {\theta }_{21}& {\theta }_{22}& {\theta }_{23}\end{array}\right]\left(\begin{array}{c}{x}_i^t\\ y_i^t\\ 1\end{array}\right)$$

(1)

where ${\theta }_{11},{\theta }_{12},{\theta }_{13},{\theta }_{21},{\theta }_{22},\mathrm{and}{\theta }_{23}$ represents the parameters required for the linear transformation, ($x_i^s$,$y_i^s$) represents the pixels sampling from the T1 segmentation label, and ($x_i^t$, $y_i^t$) represents the target coordinates of the output image ($\begin{array}{c}\to \\ \mathrm{T}1\end{array}$). Since the linear transformed pixels are not necessarily integers, we use a bilinear sampler to compensate for the missing pixels. The formula is as follows:

$$V_{\left(e,f\right)}^c={{\displaystyle \sum }}_{x^s}^H{{\displaystyle \sum }}_{y^s}^W{U}_{\left(x^s,y^s\right)}^{c}max\left(0,1-\left|x^t-e\right|\right)max\left(0,1-\left|y^t-f\right|\right)$$

(2)

In channel C, $V_{\left(m,n\right)}^c$ is the output image at ($e,f$) and $U_{\left(x^s,y^s\right)}^c$ is the input image at ($e,f$). If ($x^t$,$y^t$) is close to ($e,f$), we use bilinear sampling to add the pixel at ($x^s$,$y^s$).

Through the above operation, we randomly select five segmentation labels to participate in the training to obtain the transformed prediction of the T1 segmentation label, which is represented as $S_A\supset \left\{S_A^1,S_A^2,S_A^3\dots \dots S_A^x\right\}$. Thus, we can obtain a T1 image with a clear ROI border ($\begin{array}{c}\to \\ \mathrm{T}1\end{array}$).

3.2. Coarse Registration Network

To reduce the challenge of large differences in multimodal images, we designed a coarse registration network. Specifically, we pre-align the T1 and T2 images using the same U-net as the pixel alignment network. In this way, images with fewer differences are transmitted to the pixel alignment network, which reduces the learning difficulty of the pixel alignment network.

3.3. Pixel Alignment Network

3.3.1. Precision Registration

We obtain a spatial transformation through the pixel alignment network to align the ST1 image with the $\begin{array}{c}\to \\ \mathrm{T}1\end{array}$ image. The spatial transformation determines the pixel-to-pixel correspondence [38].

$$\mathsf{\omega }=argmi{n}_{\phi }reduse\left(\begin{array}{c}\to \\ \mathrm{T}1\end{array},ST1\circ \phi \right)+\lambda reg\left(\phi \right)$$

(3)

where image similarity $reduse\left(\begin{array}{c}\to \\ \mathrm{T}1\end{array},ST1\circ \phi \right)$ can be described as a combination of mean square (MSE) error and structural similarity (SSIM). $reg\left(\phi \right)$ is the regularization, whose objective is to ensure the smoothness of the deformation field. $\lambda$ is the regularization parameter that balances similarity and smoothness.

We concatenated the ST1 and $\begin{array}{c}\to \\ \mathrm{T}1\end{array}$ images generated by the border enhancement network and input them into the CNN to obtain the deformation field $\phi$. We put the ST1 and segmented label images into the deformation field to obtain the warped image (image A) and the warped segmented label. Through this method, the warped segmented labeled images are rich in structural information, and they will be the quality data for subsequent reverse teaching and learning. Our whole process can be expressed by the following formula:

$$w{\left(t\right)}^{\prime }=w+\phi \left(\mathrm{w}\right)$$

(4)

$\phi$ is the deformation field and $\phi \left(\mathrm{w}\right)$ indicates the offset distance. Each pixel w in the moving image after transformation can be defined as $w{\left(t\right)}^{\prime }$.

Since the pixels are not necessarily integer after transformation, linear interpolation is used to avoid discontinuity in the transformed image. The formula is as follows:

$$S\circ \phi \left(\mathrm{w}\right)={{\displaystyle \sum }}_{q\in Z\left(\phi \left(\mathrm{w}\right)\right)}N\left(q\right){{\displaystyle \prod }}_{d\in \left(x,y,z\right)}\left(1-\left|{\phi }_d\left(w\right)-q_d\right|\right)$$

(5)

where Z stands for the areas made up of nearby pixels. The projected results are smoother and more realistic as a result of this differentiable interpolation process.

3.3.2. Details of the Networks

The U-net used in this paper is modified from the famous U-net with an encoder-decoder symmetric architecture. In the process of feature extraction, the main role of the encoder is to extract the image features layer by layer, and the decoder recovers the image information layer by layer. The output feature map of the second convolution in each stage of the encoder is transmitted to the decoder through a skip connection and is channel-spliced with the output feature map of the upsampling layer in the corresponding stage of the decoder after clipping to achieve the fusion of shallow and deep information and provide more semantic information for the decoding process. As shown in Figure 5, each downsampling block is composed of two 3 × 3 convolutional layers and a 2 × 2 max pooling layer. In the last layer, we used two 3 × 3 convolutions with linear activations to obtain the final deformation field. After each convolution operation there is always batch normalization and leaky ReLU (unless otherwise stated). The channels of the U-net are set as 16, 32, 32, 64, 64, 64, 32, 32, and 16.

3.3.3. Inverse Deformation Field

In most previous deep-learning-based deformable registration methods, researchers would divide the input image into a fixed image and a moving image, and only consider a single mapping from the fixed image to the moving image, ignoring the inverse transformation [39]. In our framework, we try to eliminate the division between fixed images and moving images. We divide the input image into S and T and have to implement not only ${\phi }_{ST}$ but also ${\phi }_{TS}$. Inspired by previous inverse-consistent methods [40], we introduce the new inverse-consistent method, which does not require a complex network structure to achieve accurate registration results.

$$\phi \stackrel{decompose}{\to }{\phi }_x,{\phi }_y$$

(6)

$$\left({\phi }_x,{\phi }_y\right)\circ \phi =\left({\phi }_x^{\prime },{\phi }_y^{\prime }\right)\stackrel{combine}{\to }{\phi }^{\prime }$$

(7)

$${\phi }^{\prime }\times \left(-1\right)={\phi }^{-1}$$

(8)

where the above formula is the process of inverse deformation field formation. $\phi$ is the deformation field, ${\phi }_x$ and ${\phi }_y$ are the horizontal and vertical offsets, respectively, and $○$ is the transformation operator.

3.4. Reverse Teaching Network for Few-Shot Learning

Our reverse teaching warps some segmentation labels through a deformation field and transmits the structural information of the unlabeled images to the warped segmentation labels. Through the above operation, we can obtain additional training data (warped images and warped segmentation label pairs). Thus, these data are entered into the border enhancement network to teach the rich structural knowledge of the border enhancement network in reverse to achieve more accurate ROI boundary enhancement. The detailed process is as follows:

The reverse teaching warps several segmented, labeled images through the deformation field. Since the unlabeled images have rich structural knowledge, the warped, segmented, labeled images will have the structural knowledge of unlabeled images, which generates diverse pairs of warped images and warped segmented label images. Finally, the warped, segmented, labeled images will be input into our border enhancement network to teach our border enhancement network diverse structural knowledge, as shown in Figure 1. The warped image A and the warped segmented image are input to the border enhancement network to produce image C. Finally, the structure loss ${\ell }_{reverse}$ between image A and image C is calculated so that the border enhancement network will enhance more accurate ROI boundary information.

3.5. Objective Function

The objective function contains two components, including image similarity loss (${\ell }_{sim}$) and deformation field constraint conditions (${\ell }_{reg}$).

$$\ell \left(M,F,\phi \right)={\ell }_{sim}\left(\mathcal{H}\left(M,\phi \right),F\right)+\omega {\ell }_{reg}\left(\phi \right)$$

(9)

where F represents the fixed image, M represents the moving image, $\phi$ represents the deformation field, $\mathcal{H}$ represents the deformation process, ${\ell }_{sim}$ represents the similarity between moving and fixed images, ${\ell }_{reg}$ constrains the deformation field to make it smoother, and $\omega$ denotes the strength of the constrained deformation field.

where ${\ell }_{sim}$ consists of three losses, ${\ell }_{reverse},{\ell }_{local},\mathrm{and}{\ell }_{global}.$

$$\begin{gathered} {\ell }_{sim}\left(\mathcal{H}\left(M,\phi \right),F\right)=\alpha {\ell }_{local}\left(\begin{array}{c}\to \\ \mathrm{T}1\end{array},ST1\ast \phi \right)+\beta {\ell }_{global}\left(ST1,A\ast {\phi }^{-1}\right) \\ +\gamma {\ell }_{reverse}\left(A,C\right) \end{gathered}$$

(10)

${\ell }_{local}$ consists of MSE and SSIM, which reflects the similarity between image $\begin{array}{c}\to \\ \mathrm{T}1\end{array}$ and image A.

$$\begin{gathered} {\ell }_{local}\left(\begin{array}{c}\to \\ \mathrm{T}1\end{array},ST1\ast \phi \right)= \\ {\xi }_1\ast \frac{1}{\mathsf{\Omega }}{{\displaystyle \sum }}_{iϵ\mathsf{\Omega }}||ST1\ast \phi \left(i\right)-\begin{array}{c}\to \\ \mathrm{T}1\end{array}\left(i\right)|{|}^2+{\psi }_1\ast \left(1-\frac{\left(2{\mu }_{I_d}{\mu }_{I_f}+C_1\right)\left(2{\sigma }_{I_f{I}_d}+C_2\right)}{\left({\mu }_{I_f}^2+{\mu }_{I_d}^2+C_1\right)\left({\sigma }_{I_f}^2+{\sigma }_{I_d}^2+C_2\right)}\right) \end{gathered}$$

(11)

where ${\xi }_1$ and ${\psi }_1$ are used as the weight factors to prevent large deviations and Ω is the spatial domain. Where ${\mu }_{I_d}and{\mu }_{I_f}$ represent the local means of image $\begin{array}{c}\to \\ \mathrm{T}1\end{array}$ and image A. ${\sigma }_{I_f}\mathrm{and}{\sigma }_{I_d}$ represent the standard deviations of images $\begin{array}{c}\to \\ \mathrm{T}1\end{array}$ and image A. $C_1$ and $C_2$ are minor constants required to prevent instability.

${\ell }_{global}$ consists of PCC and SSIM, which reflects the similarity between image $ST1$ and image B.

$$\begin{gathered} {\ell }_{global}\left(ST1,A\ast {\phi }^{-1}\right)={\xi }_2\ast \left(1-\frac{\left(2{\mu }_{I_d}{\mu }_{I_f}+C_1\right)\left(2{\sigma }_{I_f{I}_d}+C_2\right)}{\left({\mu }_{I_f}^2+{\mu }_{I_d}^2+C_1\right)\left({\sigma }_{I_f}^2+{\sigma }_{I_d}^2+C_2\right)}\right)+ \\ {\psi }_2\ast \left(1-\frac{{{\displaystyle \sum }}_{i\in \mathsf{\Omega }}\left(ST1\left(i\right)-\overline{ST1}\right)\left(A\ast {\phi }^{-1}\left(i\right)-\overline{A\ast {\phi }^{-1}}\right)}{\sqrt{{{\displaystyle \sum }}_{i\in \mathsf{\Omega }}\left(ST1\left(i\right)-\overline{ST1}\right)}\sqrt{{{\displaystyle \sum }}_{i\in \mathsf{\Omega }}A\ast {\phi }^{-1}\left(i\right)-\overline{A\ast {\phi }^{-1}}}}\right) \end{gathered}$$

(12)

where ${\xi }_2$ and ${\psi }_2$ are used as the weight factors to prevent large deviations. Where ${\mu }_{I_d}and{\mu }_{I_f}$ represent the local means of image ST1 and image B. ${\sigma }_{I_f}\mathrm{and}{\sigma }_{I_d}$ represent the standard deviations of image ST1 and image B. $\overline{A\ast {\phi }^{-1}}$ and $\overline{ST1}$ stand for the average intensities.

${\ell }_{reverse}$ consists of MI and MSE, which reflects the similarity between image A and image C.

$${\ell }_{reverse}\left(A,C\right)={\xi }_3\ast \frac{1}{\mathsf{\Omega }}||C-A|{|}^2+{\psi }_3\ast \left(\iint p\left(A,C\right)\right)log\frac{p\left(A,C\right)}{p\left(A\right)p\left(C\right)}{d}_x{d}_y$$

(13)

where the adjustment factors ${\xi }_3$ and ${\psi }_3$ are used to balance the two losses. If A and C are independent, then $p\left(A,C\right)$ is equal to $p\left(A\right)p\left(C\right)$ and MI(A, C) will be 0, which means A and C are uncorrelated.

$${\ell }_{reg}={{\displaystyle \sum }}_{x\in \mathsf{\Omega }}\left|\right|\nabla h\left(x\right)|{|}^2$$

(14)

The total losses are as follows:

$$\ell =\alpha {\ell }_{local}\left(\begin{array}{c}\to \\ \mathrm{T}1\end{array},ST1\ast \phi \right)+\beta {\ell }_{global}\left(ST1,A\ast {\phi }^{-1}\right)+\gamma {\ell }_{reverse}\left(A,C\right)+\omega {\ell }_{reg}\left(\phi \right)$$

(15)

4. Experiments

4.1. Dataset

We used the IXI dataset for training, which was acquired from a Philips 1.5T system (Philips, Amsterdam, The Netherlands), Philips 3T system(Philips, Amsterdam, The Netherlands), and GE 1.5T system (GE, Boston, MA, USA). We randomly selected 450 T1 and T2 images from the IXI dataset to participate in the training. Since the IXI dataset is a 3D dataset, each data format was 256 × 256 × 130, we crop them to 224 × 224 × 130, and 130 slices were acquired. We selected slices from 64 to 67 to participate in the training. The performance of the model was validated on the Mrbrain [41] dataset. MRbrain is composed of 20 images of normal subjects acquired on a 3T MRI scanner. We randomly selected eight 3D images and cropped each image from 240 × 240 × 48 to 224 × 224 × 48, so that 48 slices were acquired. In this experiment, we selected 22 to 27 slices for registration. Each subject is labeled as eight ROIs.

The generality of this experiment is verified on three datasets, OASIS [42], CIT168 [43], and SRI24 [44]. The OASIS dataset was acquired by 1.5T Siemens equipment, which consists of 271 T1 images of the normal human brain. We resampled from 192 × 160 to 224 × 224 and then selected 100 images to participate in the registration. Each subject was labeled as 36 ROIs. For the SRI24 and CIT168 datasets, we cropped them to 224 × 224, then selected two 3D images and sliced each 3D image into 10 slices to participate in the registration.

4.2. Implementation Details

Our proposed model was implemented on the Tensorflow (version 1.14.0) framework and trained on Intel i7-10700 CPU and GeForce RTX 2080ti GPU with 11 GB memory. We used a learning rate of $1\times {10}^{-4}$ and the Adam optimizer to update the neural network weights in each network. Due to computer memory constraints, the batch size was set to eight. When the best performance is reached, the experimental training will automatically end. The whole training process takes about 10 h.

4.3. Evaluation and Validation

We use three metrics to evaluate the registration performance of the model, the percentage of non-positive values in the determinant of the Jacobian matrix over the deformation field, the DSC, and precision.

$$\mathrm{Dice}\left(\mathrm{F},\mathrm{M}\right)=\frac{2\mathrm{TP}}{2\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}\times 100\%$$

(16)

$$\mathrm{Precision}\left(\mathrm{F},\mathrm{M}\right)=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}\times 100\%$$

(17)

$$\mathrm{Recall}\left(\mathrm{F},\mathrm{M}\right)=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(18)

Ground Truth	Prediction
	True	False
True	TP (true positive)	FN (false negative)
False	FP (false positive)

TP, FP, and FN represent the number of pixels in the different cases, respectively. We compute these scores for each case separately and report the average results.

$$\mathrm{Det}.\mathrm{Jac}\left(\mathrm{i},\mathrm{j},\mathrm{k}\right)=\left|\begin{array}{ccc}\frac{\partial \mathrm{i}}{\partial \mathrm{x}}& \frac{\partial \mathrm{j}}{\partial \mathrm{x}}& \frac{\partial \mathrm{k}}{\partial \mathrm{x}}\\ \frac{\partial \mathrm{i}}{\partial \mathrm{y}}& \frac{\partial \mathrm{j}}{\partial \mathrm{y}}& \frac{\partial \mathrm{k}}{\partial \mathrm{y}}\end{array}\right|$$

(19)

$$\mathrm{Npr}.\mathrm{Jac}=\frac{\sum \mathsf{\sigma }\left(\mathrm{Det}\left(\mathrm{J}\left(\mathrm{i},\mathrm{j},\mathrm{k}\right)\right)\le 0\right)}{\mathrm{n}}$$

(20)

$\mathrm{Det}\left(\mathrm{J}\right)=1$ indicates that no volume behavior has occurred, where $\mathrm{Det}.\mathrm{Jac}$ represents Jacobian determinant. Det (J) > 1 indicates that the point is expanding. 0 < Det (J) < 1 indicates that the point is contracting. Det (J) ≤ 0 indicates a folding phenomenon. $\mathsf{\sigma }$ is a linear activation function. n represents the total number of elements.

5. Results

In the registration task, we compared our model with the traditional method SyN [1] and three deep-learning-based methods, VoxelMorph [22], VM-diff (VoxelMorph-diff) [45], and Cof-net [46]. Pre-net is the model trained by removing a reverse teaching network on the base of reverse-net and using 40 segmentation labels. Un-reverse-net is the model without labels involved in training (it participates in training with T2 images instead of segmented label images). We validated the performance and generalization ability of the model with four brain MR datasets.

5.1. Border Enhancement Experiment

Our reverse teaching network utilizes the deformation field to generate additional training data (warped images and warped segmentation label pairs), which can reversely teach rich structural knowledge to the border enhancement network. The above operation makes the border enhancement network enhance more accurate ROI boundaries and provide quality data for the subsequent pixel alignment network. As shown in Figure 6, the ROI boundary enhancement results show a significant visual improvement, as reflected by finer and clearer borders with fewer blurred structures. The enlarged area of the MRI in the brain showed complete border enhancement areas with clear borders. Our ROI border enhancement network enhanced the ROI boundary information of the image and effectively suppressed the misalignment due to boundary blurring.

5.2. Ablation Experiment

We devised ablation experiments to validate the contribution of reverse teaching networks. We conducted experiments using brain MR data to evaluate the registration performance of the model using the average DSC, the percentage of non-positive values in the determinant of the Jacobi matrix of the deformation field, and the average alignment accuracy per image.

The results of the ablation experiments are shown in

Table 1. Quantitative results of ablation experiments. The numbers following the ± indicate the standard deviation.

Method	DSC (%)	Precision (%)	Det.Jac (%)
Cof-net [46]	81.70 ± 3.71	90.59 ± 3.05	0.006 ± 0.002
Pre-net	84.44 ± 0.67	89.94 ± 4.27	0.002 ± 0.001
Un-reverse-net	83.29 ± 0.15	89.45 ± 1.23	0.004 ± 0.012
Reverse-net	85.20 ± 1.18	94.41 ± 0.89	0.004 ± 0.001

. The structural information of different unlabeled images is passed to the warped, labeled images through the deformation field formed by the pixel alignment network, and image A and the warped, labeled images with rich structural knowledge will provide this rich structural knowledge to the reverse teaching network. The above operation can provide quality data for the subsequent pixel alignment network. Compared with Pre-net, the structural information brings 0.76% DSC and 4.47% precision improvement to our model. Compared with Cof-net, the structural information brings 3.5% DSC and 3.82% precision improvement to our model. In addition, as shown in Figure 7. Compared with the other two models, our model is richer in detail. The above results show that the reverse teaching network improves the registration performance.

In addition, we investigated the effect of different numbers of segmented labels on the registration performance. As shown in Figure 8. When having one segmented label, the model decreases 2.33% on DSC compared with Pre-net. When having three labels, the model improves 0.41% on DSC compared with Pre-net. This shows that our reverse teaching network can improve the registration performance of the model. As the number of labels increases, our model is further enhanced. When the number of labels is increased to five, the reverse-net improves 0.76% on DSC compared with Pre-net. This further shows that our network has improved the ability of the border enhancement network sensing boundary through reverse teaching.

Finally, we evaluated the effect of the model without segmented label participation. The results of quantitative analysis are shown in Table 1. Compared to Cof-net, our model improves 1.59% on DSC and decreases 0.002 on Det.Jac. Compared to VoxelMorph, our model improves 2.45% on DSC and decreases 0.002 on Det.Jac. The above results show the superiority of our model design. When labels are involved, the model results are even better.

5.3. Comparative Experiments

We compared our method with four state-of-the-art registration methods, including symmetric normalization (SyN) and three deep-learning-based methods, VoxelMorph, VM-diff, and Cof-net. On the same datasets, we trained deep learning methods with recommended hyper-parameters from scratch. We quantitatively evaluated the performance from four perspectives: Precision, DSC, Runtime, and Det.Jac.

Table 2. Quantitative results on the MRbrain dataset. Bold indicates the highest score.

Method	DSC (%)	Precision (%)	Runtime (s)/Slice	Det.Jac (%)
SyN [1]	79.48 ± 0.78	84.07 ± 0.86	2.034	0.071 ± 0.007
Voxelmorph [22]	80.84 ± 0.78	91.10 ± 1.18	0.558 ± 0.017	0.006 ± 0.002
VM-diff [45]	79.59 ± 2.31	86.68 ± 2.52	0.423 ± 0.011	0.013 ± 0.005
Cof-net [46]	81.70 ± 3.71	90.59 ± 3.05	0.367 ± 0.013	0.006 ± 0.002
Reverse-net	85.20 ± 1.18	94.41 ± 0.89	0.359 ± 0.011	0.004 ± 0.001

summarizes the experimental results on the MRbrain dataset. It can be seen that our method obtains better mean DSC scores and lower running times while achieving the lowest number of folds. In the MRbrain dataset, compared with VoxelMorph, the DSC score and Precision score of our model increased by 4.36% and 3.31%, respectively. The Det.Jac indexes of our model are decreased from 0.006% to 0.004%. Compared with Cof-net, our model improves 3.5% in the DSC metric and 3.82% in Precision, and Det.Jac decreases from 0.006% to 0.004%. At the same time, our model has a lower average processing time per slice compared to Voxelmorph, VM-diff, and Cof-net. In addition, it can also be seen from the boxplot in Figure 9 that our model achieves the best results among the three evaluation metrics.

Figure 10 shows the visualization of the images from the perspective of axial slices. The columns marked with W indicate the warped images. φ indicates the deformation field visualization figure. The red color in the visualized image indicates the horizontal transformation and the green color indicates the vertical transformation. The higher the red or green signal, the larger the transformation. As shown in Figure 10, typical registration results from the MRbrain dataset show that the warped image generated by our method is most like the fixed image; our method can achieve the better registration among several methods.

5.4. Evaluation and Analysis on CIT168, SRI24, and OASIS

In order to evaluate the generalizability of the network, many experiments were performed on images from different scanners. It is worth noting that the proposed method uses the same model as the comparative experimental model. The experiments were performed on the CIT168, SRI24, and OASIS datasets. As shown in

Table 3. Quantitative results on OASIS, CIT168, and SRI24 datasets. Bold indicates the highest DSC score.

	Method	SyN [1]	VoxelMorph [22]	VM-Diff [45]	Reverse-Net
Dataset
OASIS [42]		67.7 ± 2.9	74.9 ± 13.7	75.2 ± 13.9	79.1 ± 1.3
SRI24 [44]		70.3 ± 5.6	77.9 ± 2.4	78.6 ± 2.9	82.6 ± 0.6
CIT168 [43]		67.8 ± 2.6	73.4 ± 3.9	73.2 ± 3.1	80.5 ± 1.9

, compared to VoxelMorph, the proposed method improved 7.3%, 4.7%, and 4.2% on the CIT168, SRI24 and OASIS datasets, respectively. As shown in Figure 11, our model shows more details on the CIT168, SRI24, and OASIS datasets. The above results show that our method achieves the best registration among several methods.

6. Discussion

For supervised registration, the number of labels limits the registration performance. Unsupervised registration is easily disturbed by outliers and task-irrelevant areas due to the constraint of missing labels, which can lead to eventual misalignment and distortion. To solve these problems, we propose a few-shot deformable image registration method based on reverse learning, which can improve registration performance effectively. For few-shot learning, the reverse teaching network utilizes the deformation field generated by the pixel alignment network to generate additional training data from different unlabeled images and teaches the border enhancement network rich structural knowledge for more accurate enhancement of T1 image ROI boundaries. Therefore, the above operation can provide high-quality data for the subsequent pixel alignment network and improve the registration performance of the model. The registration results in Figure 6 also show the ability of the border enhancement network to enhance the ROI boundaries,;our experiments on brain MRI images show that our network has better registration performance and effective texture preservation. Compared with Cof-net, we achieved 3.5% improvement in DSC while reducing the processing time per slice, which indicates that our framework has better potential for future clinical applications.

Reverse-net has improved deformable medical image registration with only a few segmentation labels, but the number of segmented labels becomes a challenge. In order to fully utilize the segmentation labels, we introduced the reverse teaching network and redesigned the loss function. Experiments in which different numbers of segmentation labels are used are conducted on the MRbrain dataset. It was shown that competitive registration accuracy could be obtained by using only five segmentation labels.

Although methods based on few-shot learning are efficient, the smoothness of the deformation field is insufficient. In order to enhance the smoothness, various joint loss functions were designed and introduced, and suitable hyperparameters were selected after extensive experiments. As can be seen in Figure 10, the smoothness of the deformation field is better with our method.

By redesigning the network structure and loss function, our model shows better performance on brain datasets. However, MR–CT multimodal registration based on few-shot learning is still a challenging task in medical image deformation registration, so future research applying our reverse-net for MR–CT multimodal registration tasks is significant.

7. Conclusions

In this paper, we propose a deformable medical image registration model based on few-shot learning, which can provide doctors with a reliable diagnosis and reduce misdiagnosis rates. Different from other methods, we introduced the reverse teaching network, which can reduce the number of segmentation labels while transmitting rich structural knowledge for the border enhancement network. The above operations can make the ROI boundaries of T1 images clearer which provides better quality data for the subsequent pixel alignment network and improve the registration performance of the model. The smoothness of the deformation field was improved by introducing a new loss function and redesigning the individual weights of the loss function. The method obtained satisfactory registration results on the brain dataset. In addition, the experimental results on the unseen data also show that our model has better generalizability. MR–CT registration remains a challenging task in the field of medical image registration due to the lack of MR–CT data. Therefore, in the future, we will apply the reverse-net to MR–CT multimodal registration task based on few-shot learning.