nmPLS-Net: Segmenting Pulmonary Lobes Using nmODE

Abstract

Pulmonary lobe segmentation is vital for clinical diagnosis and treatment. Deep neural network-based pulmonary lobe segmentation methods have seen rapid development. However, there are challenges that remain, e.g., pulmonary fissures are always not clear or incomplete, especially in the complex situation of the trilobed right pulmonary, which leads to relatively poor results. To address this issue, this study proposes a novel method, called nmPLS-Net, to segment pulmonary lobes effectively using nmODE. Benefiting from its nonlinear and memory capacity, we construct an encoding network based on nmODE to extract features of the entire lung and dependencies between features. Then, we build a decoding network based on edge segmentation, which segments pulmonary lobes and focuses on effectively detecting pulmonary fissures. The experimental results on two datasets demonstrate that the proposed method achieves accurate pulmonary lobe segmentation.

1. Introduction

Pulmonary lobe segmentation plays a crucial role in clinical practice, including assisting in diagnosing pulmonary diseases and planning lung surgeries. Relying on manual pulmonary lobe segmentation is time-consuming and labor-intensive. Furthermore, manual segmentation results can be influenced by subjective factors. Thus, an effective automatic method is vital to ensure high efficiency and repeatability.

Although automatic pulmonary lobe segmentation algorithms have made significant progress, there are still several challenges in their practical clinical application. Regarding right pulmonary lobe segmentation, it comprises three lobes and is more complex than the left ones. Specifically, the right middle lobe is sandwiched between two other lobes. Moreover, segmenting pulmonary fissures always exhibit another challenge for unclear or incomplete fissures. The existing methods rarely consider the crucial aspect of learning correlations between features and cannot effectively extract the overall information between samples, which is essential for resolving the abovementioned issue.

To address these issues, this study proposes a novel pulmonary lobe segmentation method using nmODE [1], called nmPLS-Net. Firstly, this method benefits from the robust nonlinear expressive and memory capabilities of nmODE, and the constructed model can extract more complex and rich pulmonary features to support the precise segmentation of pulmonary lobes. Then, due to the significant challenges in identifying pulmonary fissures, nmPLS-Net leverages a fissure-enhanced associative learning approach, focusing on learning detailed features of pulmonary fissures. Finally, this study utilizes two datasets to evaluate the proposed method’s effectiveness. The experimental results demonstrate that the proposed method achieves competitive results compared to other state-of-the-art methods. The main contributions of this study can be summarized as follows:

• The main contribution of this study is the combination of nmODE with convolutional neural networks in the task of pulmonary lobe segmentation. This combination leverages nmODE’s high nonlinearity and memory capabilities better to identify global pulmonary features and the relationships between these features;
• This study employs a novel fissure-enhanced associative learning approach to make the model focus on detail identification near pulmonary fissures;
• This study conducted experiments on two datasets, and our network achieved competitive overall segmentation results, significantly outperforming previous works in segmenting the right middle pulmonary lobe.

2. Related Works

2.1. Medical Image Processing

Deep learning has found widespread application in medical image processing due to its simplicity and efficiency. Early deep learning image segmentation networks have used fully convolutional networks (FCNs) [2], demonstrating good performance due to their ability to extract hierarchical features and end-to-end processing of inputs of arbitrary sizes. However, an FCN’s limitation lies in the fact that it under-utilizes features at different resolutions obtained from different layers, especially in large-resolution medical images. U-Net [3], with its multi-level encoder-decoder architecture and skip connections, addresses the FCN’s limitations and has gained popularity in medical image processing. Shoaib et al. [4] used several different state-of-the-art convolutional neural networks to perform left ventricle segmentation, achieving commendable results. Zhao [5] and his team conducted a detailed investigation and analysis of deep learning methods for a neuroimaging diagnosis of Alzheimer’s disease.

The introduction of the transformer [6] architecture had a profound impact on the field of deep learning. Based on the transformer, the vision transformer (ViT) has found widespread application in image processing. The swin-transformer [7] proposed by Liu et al. addressed the computational challenges associated with ViT in image processing by incorporating a sliding window and partial attention mechanism. The swin-transformer v2 [8] further improved upon the swin transformer. TransUNet [9], developed by Chen et al., combines the strengths of U-Net and the transformer to process medical images. Shen et al. [10] proposed a boundary-guided transformer that can simultaneously identify rectum and tumor regions in sagittal MR images of rectal cancer patients. For a comprehensive overview of related research, Liu et al. offered a survey [11]. Transformer-based approaches have shown superior performance in many aspects. However, due to its huge parameters, traditional 3D convolutional neural networks are currently more practical in this study.

Pulmonary lobe segmentation typically uses the entire CT scan as input. Three-dimensional U-Net structures are commonly employed for this task, such as V-Net [12]. Based on V-Net, Imran et al. [13] introduced a progressive dense V-Net, enhancing performance through feature fusion. Ferreira et al. [14] proposed FRV-Net, utilizing regularization techniques to achieve better segmentation results, even with limited data. Tang et al. [15] employed a mixed loss function to enhance the model’s focus on challenging pixels in pulmonary lobes. While the previously mentioned methods have achieved good results, they often have a large number of parameters. To obtain a more efficient network, Lee et al. [16] introduced an efficient PLS-Net for pulmonary lobe segmentation. They leveraged 3D depth-wise separable convolution to effectively reduce parameters while employing dilated residual modules to enhance the network’s receptive field without increasing the parameter count. Despite significantly reducing the model’s parameter count and improving the receptive field, PLS-Net exhibits subpar performance at the pulmonary lobe edges and with incomplete pulmonary fissures. To address the issue of incomplete fissures, Xie et al. [17] harnessed relational modeling for pulmonary lobe segmentation, effectively capturing relationships and contextual information among pulmonary lobes. However, this approach comes at the cost of a substantially higher parameter count than PLS-Net, resulting in relatively slower computational speeds due to its intricate process. Fan et al. [18] introduced a learnable interpolation and extrapolation network designed to handle incomplete pulmonary fissures and enhance segmentation performance. Nevertheless, this method necessitates five processing stages, including lung segmentation, pulmonary fissure segmentation, pulmonary fissure completion, pulmonary lobe segmentation, and post-processing, significantly diminishing its efficiency. Liu et al. [19] took the enhancement of PLS-Net a step further with RPLS-Net by incorporating pulmonary lobe boundary segmentation as an auxiliary task, thus focusing on challenging edge pixels. This approach achieved superior results compared to PLS-Net without significantly increasing the parameter count. However, it is worth noting that RPLS-Net cannot learn correlations between features and heavily relies on an expanded receptive field to gather global information. This limitation leads to relatively poorer performance on the right middle pulmonary lobe compared to other pulmonary lobes.

2.2. Neural Ordinary Differential Equations

In recent years, deep learning has made remarkable strides across various applications. One of the most renowned neural networks in the realm of deep learning is Res-Net [20], celebrated for its robustness due to the incorporation of residuals. Research conducted by Haber et al. [21] has revealed that Res-Net can be viewed as a discrete counterpart of certain differential equations. The concept of neural ordinary differential equations (NODEs) was introduced by Chen et al. [22], shifting the paradigm from discrete neural networks to continuous differential equations. The core concept in Res-Net, the residual, can be likened to a discrete differential equation form. Denote the output of the l-th layer in a neural network as $x^l$, conventionally. In that case, the output of the subsequent layer ($l+1$) can be represented as $x^{l+1}=f\left(x^l\right)$, where f encompasses the computational processes of a single layer, including linear transformations, regularization, and nonlinear activation functions. In the case of Res-Net, the output of the $(l+1)$-th layer is defined as $x^{l+1}=x^l+f\left(x^l\right)·1$. If considering l as time step t and interpreting the output $x^l$ as a function $y\left(t\right)$ evolving over time steps, with the step interval of 1 being analogous to a time interval $\Delta t$, the model’s computations can be conceptualized as the equation $y(t+\Delta t)-y\left(t\right)=f\left(y\right(t\left)\right)·\Delta t$. As the number of layers in the model approaches infinity, i.e., as $\Delta t$ tends to zero, it arrives at the NODE expression, a differential equation illustrated in Equation (1). Obtaining the exact solution to this equation is nearly impossible. In practice, by providing an initial value $y\left(0\right)$ to the abovementioned differential equation, an ODE solver library can be leveraged to approximate the solution as $y\left(T\right)$.

$$\dot{y}\left(t\right)=\underset{\Delta t\to 0}{lim}\frac{y(t+\Delta t)-y\left(t\right)}{\Delta t}=f\left(y\left(t\right)\right).$$

(1)

Compared to traditional neural networks, neural ordinary differential equations exhibit more nonlinearity and fewer parameters. Subsequently, many related studies have emerged. Dupont et al. [23] pointed out that the NODE, when using the data input as the initial value, preserves the spatial structure of the input, and the authors provided examples illustrating the NODE’s limitations in this regard. However, this issue can be addressed by inputting the data through external input variables instead of using them as initial values for the differential equation [1]. For more related methods, please refer to Refs. [24,25].

3. Methods

3.1. Problem Formulation

The main task of this study is a multi-class segmentation problem. The input of the constructed model is a whole CT image, denoted by $\mathit{X}$$\in R^{1\times D\times H\times W}$, and the labels are denoted by $\mathit{G}$$\in R^{1\times D\times H\times W}$. $\mathit{D}$, $\mathit{H}$, and $\mathit{W}$ denote the depth, height, and width in an input image, respectively. $\mathcal{F}$ denotes the constructed model. $\mathcal{P}$ and ${\mathcal{P}}_e$ denote the output of the main and fissure-enhanced associative learning task. $\mathcal{L}$ denotes the loss function to optimize the constructed model.

3.2. Model

The architecture of the proposed model in this study was inspired by PLS-Net [16] and RPLS-Net [19]. Figure 1 illustrates the workflow of the proposed method. The reason for choosing PLS-Net as the backbone network for this study is that, compared to commonly used backbone networks for pulmonary lobe segmentation, such as V-Net, PLS-Net has fewer parameters and yields better results. Additionally, using the same backbone network and dataset as in Ref. [19] ensures a fair and accurate comparison of the performance of the nmODE and FEAL modules.

Figure 1. The detailed structure of nmPLS-Net, the modules of the backbone network encoder, and the detailed structure of the nmODE module are displayed on the right. The color coding for each module is as shown in the figure. Within the DRDB module, DRDB × 1, DRDB × 2, and DRDB × 4 represent one, two, and four consecutive DRDB modules, respectively. Encoder1, Encoder2, and Encoder3 in the figure are denoted as $E1$, $E2$, and $E3$ in this paper. Similarly, Decoder1, Decoder2, and Decoder3 are denoted as $D1$, $D2$, and $D3$.

The model inputs the original CT image $\mathit{X}$ and then produces pulmonary lobe result $\mathcal{P}$ and fissure-enhanced associative learning result ${\mathcal{P}}_e$. This process can be represented as $\mathcal{P},{\mathcal{P}}_e=\mathcal{F}\left(\mathit{X}\right)$.

The model’s encoder consists of three hierarchical stages, denoted as $E1$, $E2$, and $E3$. Similarly, the decoder comprises three corresponding stages, namely $D1$, $D2$, and $D3$, and a classifier for generating results. Skip connections connect the same stage.

The model’s encoder comprises Dilated Separate Convolutions (DS Conv) and Dilated Residual Dense Block (DRDB). In the case of dilated separate convolution, its principle is similar to regular depth-wise separable convolution [26]. Additionally, dilated separate convolution employs dilated convolutions [27] to increase the receptive field, thus enlarging the receptive field without increasing the parameter count. The Dilated Residual Dense Block, written as DRDB for simplicity, used in the model consists of dilated separate convolutions and their structure, as shown in Figure 1. This module combines the results of convolutions with different receptive fields in a residual manner to achieve a wide range of receptive field combinations. In each encoder stage, the dilation rate for the dilated separate convolution is set to 1, indicating no dilation, and the convolution stride is set to 2 for down-sampling. In the DRDB module, the dilation rates for the four dilated separate convolutions are set from top to bottom as $(1,2,3,4)$, as recommended in [16]. The input for each encoder stage is concatenated with the original image down-sampled to the corresponding size to compensate for the information loss during down-sampling. In stage $E3$, the nmODE module is introduced to process the features extracted by its DRDB module. The input to the nmODE module is the output feature of the DRDB module, and the output is the solution of the differential equation in the nmODE module. The following section will provide a detailed description of the nmODE module.

The model’s decoder consists of Dilated Separate Convolutions (DS Conv) and Depth-wise Separable Deconvolutions (DS DeConv). The structure of dilated separate convolution is the same as that used in the encoder, with a dilation rate of 1. Depth-wise separable deconvolution employs deconvolution to perform depth-wise separable convolution for up-sampling. The outputs from the corresponding layers of the encoder, transmitted through skip connections, are used to supplement the missing feature information after up-sampling. The classifier comprises two $1\times 1\times 1$ convolution modules responsible for generating segmentation results from model outputs. Within the classifier, a fissure-reinforcement method is introduced, with further information available in Section 3.4.

3.3. Neural Memory Ordinary Differential Equation (nmODE)

Neural differential equations exhibit a higher degree of nonlinearity and better robustness, and the architecture of the NODEs is depicted in Figure 2a. However, regular NODE has some limitations, such as their inability to represent mappings such as $g\left(1\right)=-1,g(-1)=1$. This limitation primarily arises since NODE treats data as initial values, which preserves the spatial structure of the input data [23]. Furthermore, differential equations can describe dynamical systems, and research suggests that attractors within dynamical systems are considered to be associated with memory capacity [28,29]. However, traditional neural differential equations often fail to harness the memory capacity offered by attractors.

Figure 2. Architecture of NODE and nmODE. (a) represents the typical NODE architecture, where the initial value of the differential equation, denoted as $y\left(0\right)$, is derived from the data itself or the output of the previous layer, denoted as $x^l$. The output is the numerical solution of the differential equation, denoted as $x^{l+1}$. (b) represents the nmODE Architecture, where the initial value of the differential equation, denoted as $y\left(0\right)$, is a fixed value of 0. Data or the output of the previous layer serves as an external input $\mathit{\gamma }$ into the module. The output is the same as in typical NODE, which is the numerical solution of the differential equation.

Yi [1] proposed a new type of neural differential equation, called the neural memory differential equation (nmODE), which not only addresses the inherent limitations of traditional differential equations but also fully harnesses the memory capacity offered by dynamical systems. The nmODE treats the input data as external parameters rather than utilizing them as the initial conditions for the ordinary differential equation while employing a fixed initial value. By doing so, nmODE not only avoids the problem mentioned above but also separates the functionality of neurons into learning and memory components. Learning only occurs in the learning part, while the memory part is responsible for mapping the input to its global attractor, establishing a mapping from the input space to the memory space [1]. The structure of nmODE is illustrated in Figure 2b, where $\mathit{\gamma }$ represents external input, and the learning part occurs within the transformation $g\left(x^l\right)$. In our model, $\mathit{\gamma }$ represents the output of the DRDB module within the encoder stage, labeled as $E3$. The formula representation of the nmODE architecture is shown in Equation (2). Based on the nmODE architecture, Yi [1] also proposed a novel and efficient implementation, which is structured as shown in Equation (3). Furthermore, Yi demonstrated in Ref. [1] that a key property of this implementation is that it has a unique global attractor for each input, defining a mapping that transforms an external input x into an output $y\left(T\right)$ within the memory space.

$$\left\{\begin{array}{c}\gamma =g\left(x^l\right),\hfill \\ \dot{y}\left(t\right)=f(y\left(t\right),t,\gamma ).\hfill \end{array}\right.$$

(2)

$$\dot{y}\left(t\right)=-y\left(t\right)+{sin}^2(y\left(t\right)+\gamma ).$$

(3)

In pulmonary lobe segmentation, the right lung with three lobes is more challenging than the left lung, and cases with unclear or incomplete boundaries between pulmonary lobes can make segmentation even more difficult. Human annotators rely on the overall pulmonary structure and dependencies between objects when performing manual segmentation. Similarly, learning better global features and capturing dependencies between features can help address these challenges for neural network models.

In this study, an nmODE module was embedded in the encoder stage $E3$ to further refine the features extracted by the encoder. The powerful nonlinear expressive capability of nmODE enables effective learning of feature representations in complex conditions. Embedding the nmODE module in PLS-Net leverages nmODE’s memory capacity and strong nonlinear capabilities to obtain better global features and learn the relationships between features. Across different samples, similar features are mapped to nearby attractors in the nmODE memory space, allowing nmODE to memorize similar information between samples and enabling the model to capture overall pulmonary characteristics. The nmODE module can simultaneously learn and remember dependencies between features during the mapping process. For individual samples, nmODE’s high nonlinearity and memory capabilities refine and fuse information from different channels of features more effectively, resulting in more global feature information. The primary reasons for placing the nmODE module in the last stage of the encoder are two-fold. Firstly, the lowest-level stage boasts the largest channel dimension and the most expansive receptive field. A higher channel dimension facilitates a more comprehensive exploration of profound connections among features. Moreover, a larger receptive field diminishes interference from superfluous fine-grained details and steers the input features toward a more globally contextualized state within the nmODE module. Second, it is for performance considerations. The lowest-level output has the lowest resolution, and placing the nmODE module here does not significantly increase memory usage or reduce model computation speed during training.

3.4. Fissure-Enhanced Associative Learning

Previous work [19] proposed an auxiliary task involving segmenting the entire pulmonary lobe edge. Complete pulmonary segmentation is relatively straightforward, and our observations during training have shown that the network tends to distinguish the outer pulmonary contour first naturally. So, emphasizing the model’s attention on external boundaries is unnecessary. Instead, this study aim to direct the model’s focus toward the information within the pulmonary fissures, which are more challenging areas.

This study introduces a novel approach called Fissure-Enhanced Associative Learning (FEAL) to encourage the model to focus on voxels near pulmonary fissures and integrate pulmonary fissure information during classification. The associated loss is computed using the pulmonary fissure segmentation results as a secondary model output. This integration enables the model to identify voxels corresponding to pulmonary fissure locations accurately. Within the classifier, the FEAL module merges the pulmonary fissure segmentation output with the decoder’s output, employing a convolutional layer to combine the fissure information with the decoder’s output features and make corresponding predictions. Previously, in multitask learning approaches for pulmonary lobe segmentation, the model primarily relied on auxiliary tasks to encourage the decoder to prioritize pulmonary fissures. However, during the final classification step, the classifier independently handled pulmonary fissure segmentation. In contrast, our fissure-enhanced associative learning approach incorporates classification information during the classification process, further associating decoder features with pulmonary fissure information.

A simple and efficient algorithm can be used to automatically generate pulmonary fissure labels from the label mask of CT images without any manual intervention. Thanks to the fact that the values in CT label masks are limited to a few categories (e.g., a pulmonary lobe segmentation mask may have only 6 classes, with values ranging from 0 to 5 as integers), we determine whether a point belongs to the pulmonary fissure by sliding a window and verifying whether the values inside the window belong to two neighbor target categories simultaneously. The process is outlined in pseudocode in Algorithm 1. The pulmonary fissure segmentation results obtained by the algorithm are shown in Figure 3. This algorithm only needs to scan the image once and has a time complexity of $\mathit{O}\left(\mathit{n}\right)$, assuming the input size is $\mathit{n}$. Compared to the methods used in previous work, which involved image processing with the scikit-image library, Gaussian filtering, and thresholding, this algorithm is simpler and more efficient. Additionally, it can mitigate the adverse effects of partial mislabeling. For example, in Figure 3, a small portion of the right upper lobe is incorrectly labeled as the left lower lobe, which is ignored in the generated pulmonary fissure mask.

Algorithm 1 Pulmonary Fissure Label Generation Algorithm

1: $\mathrm{neighbours}\leftarrow \left[\right(1,2),(1,3),(2,3),(4,5\left)\right]$ 2: $\mathrm{img}\_\mathrm{array}\leftarrow \mathrm{CT}\mathrm{image}\mathrm{mask}\mathrm{of}\mathrm{shape}[\mathit{D},\mathit{W},\mathit{H}]$ 3: $\mathrm{result}\leftarrow \mathrm{metric}\mathrm{with}\mathrm{all}\mathrm{False}\mathrm{values}\mathrm{of}\mathrm{shape}[\mathit{D},\mathit{W},\mathit{H}]$ 4: for $i\leftarrow 1$ to $\mathit{D}$, $j\leftarrow 2$ to $\mathit{W}-1$, $k\leftarrow 2$ to $\mathit{H}-1$ do 5:    $\mathrm{window}\leftarrow \mathrm{img}\_\mathrm{array}[i,j-1:j+1,k-1:k+1]$ 6:    for $\mathrm{pair}\in \mathrm{neighbors}$ do 7:          if $\mathrm{pair}\left[1\right]\in \mathrm{window}$ and $\mathrm{pair}\left[2\right]\in \mathrm{window}$ then 8:                $\mathrm{result}[i,j,k]\leftarrow \mathrm{True}$ 9:          end if 10:    end for 11: end for 12: Output: Pulmonary fissure labels.

Figure 3. (a) Lung CT image and its mask, with an annotation error circled in blue. (b) Pulmonary fissures separated by Algorithm 1. In this visual representation, distinct lung regions are color-coded: yellow for the right upper lobe, green for the right middle lobe, brown for the right lower lobe, blue for the left upper lobe, red for the left lower lobe, and white for pulmonary fissures.

3.5. Loss Functions

The loss function used for training is a mixed loss function consisting of two components: ${\mathcal{L}}_{main}$ for calculating pulmonary lobe segmentation loss and ${\mathcal{L}}_e$ for calculating the FEAL method loss. A hybrid loss function consisting of dice loss ${\mathcal{L}}_{Dice}$ and focal loss ${\mathcal{L}}_{Focal}$ is used as the primary loss ${\mathcal{L}}_{main}$. The calculation formula for the ${\mathcal{L}}_{Dice}$ in the hybrid loss is as in Equation (4). In this equation, C represents the number of pulmonary lobe categories, and N represents the number of points in a single sample. $g_c^i$ and $p_c^i$, respectively, denote the ground truth and the probability of point $\mathit{i}$ being predicted as category $\mathit{c}$. $\mathit{ϵ}$ is the smoothing coefficient; in this study, the value is set to 1. The dice loss is calculated separately for each category. However, there is a significant difference in the number of foreground and background pixels for each category. Therefore, the focal loss is introduced to balance the positive and negative samples. The formula for the ${\mathcal{L}}_{Focal}$ used in the hybrid loss is as in Equation (5). In this equation, C, N, $g_c^i$, $p_c^i$ are the same as in Equation (4). ${\alpha }_c$ represents the weight for each category, and $\beta$ is a hyperparameter. ${\mathcal{L}}_{main}$ is as shown in Equation (6), where ${\lambda }_1$ and ${\lambda }_2$ are the weight coefficients for the two types of losses.

$$\left\{\begin{array}{c}{I}_c=\sum _{i=1}^N{g}_c^i\ast p_c^i,\hfill \\ U_c=\sum _{i=1}^N(g_c^i+p_c^i),\hfill \\ {\mathcal{L}}_{Dice}(\mathcal{P},G)=\sum _{c=1}^{C}1-\frac{2I_c+ϵ}{U_c+ϵ}.\hfill \end{array}\right.$$

(4)

$$\begin{array}{cc}\hfill {\mathcal{L}}_{Focal}(\mathcal{P},G)=& -\sum _{c=1}^C{\alpha }_c\sum _{i=1}^N(g_c^i{(1-p_c^i)}^{\beta }log{p}_c^i\hfill \\ \hfill & +(1-g_c^i){\left(p_c^i\right)}^{\beta }log(1-p_c^i)).\hfill \end{array}$$

(5)

$${\mathcal{L}}_{main}(\mathcal{P},G)={\lambda }_1{\mathcal{L}}_{Dice}(\mathcal{P},G)+{\lambda }_2{\mathcal{L}}_{Focal}(\mathcal{P},G).$$

(6)

Since positive samples account for only a small portion of the pulmonary fissure labels, leading to severe class imbalance, the focal loss is used as the loss function for the fissure-enhanced associative learning output ${\mathcal{L}}_e$ to address this issue. The formula is as in Equation (7). In this context, ${\mathcal{P}}_e$ and $G_e$ represent the predictions and ground truth for pulmonary fissure, while $p_e^i$ and $g_e^i$ represent the predictions and ground truth for each pixel. ${\alpha }_e$ and $\beta e$ are hyperparameters.

$$\begin{array}{cc}\hfill {\mathcal{L}}_e({\mathcal{P}}_e,G_e)=& \sum _{i=1}^N{\alpha }_e(g_e^i{(1-p_e^i)}^{{\beta }_e}log{p}_c^i\hfill \\ \hfill & +(1-g_e^i){\left(p_e^i\right)}^{{\beta }_e}log(1-p_e^i)).\hfill \end{array}$$

(7)

The overall loss function is as in Equation (8), where $\lambda$ represents the weight of the fissure-enhanced associative learning loss function.

$$\mathcal{L}(\mathcal{P},G,{\mathcal{P}}_e,G_e)={\mathcal{L}}_{main}(\mathcal{P},G)+\lambda {\mathcal{L}}_e({\mathcal{P}}_e,G_e).$$

(8)

4. Experiments

4.1. Experimental Settings

Dataset: A base dataset consistent with Ref. [19] was used in this study, which comprises 32 chest CT scan images with a maximum slice thickness of 1 mm. The pulmonary fissure labels used for fissure-enhanced associative learning were generated from pulmonary lobe labels using the fissure label generation algorithm. Experiments were conducted on another publicly available LUNA16 dataset, as presented in Ref. [15], which contains 50 cases with unclear or incomplete pulmonary fissures and lesions to validate the effectiveness of our method. All CT data were resized to a (1 mm, 1 mm, 1 mm) spacing and filtered to include voxels with Hounsfield Unit (HU) values between 400 and 1000. The data were then normalized. A 20-voxel-wide region of uninformative background near each CT image’s boundary was cropped to improve training efficiency. For example, a CT of size $(300,512,512)$ was cropped to $(300,472,472)$ after this process.

Model and Training: Our model and training code were implemented using the PyTorch library [30]. The code of the model can be found at https://github.com/EdewagaPoe/nmPLS-Net-Segmenting-Pulmonary-Lobes-using-nmODE.git accessed on 4 November 2023. We employed PyTorch’s built-in automatic mixed precision training (AMP) to save GPU memory during training. Due to the non-uniform input sizes, the batch size was fixed at 1. The optimizer used was the Adam optimizer [31] with a learning rate of $0.001$ and a weight decay of $5\times {10}^{-4}$. The dilated residual convolution module of the model used dilation rates and growth rates of $(1,2,3,4)$ and 12, respectively, following the recommendations in Ref. [16]. The hyperparameters in the overall loss function, namely ${\lambda }_1$, ${\lambda }_2$, and $\lambda$, were set to $0.6$, $0.4$, and $0.2$, respectively. ${\alpha }_c$ and ${\alpha }_e$ were both set to 1, and $\beta$ and ${\beta }_e$ were set to 2. The training was performed on an NVIDIA TITAN RTX GPU with 24 GB of VRAM. We trained the model for 100 epochs and selected the best-performing result on the validation set for testing.

Evaluation Metrics: This study used the dice coefficient (DC) as an evaluation metric for segmentation performance, as shown in Equation (9). Here, ${\mathcal{P}}_{val}$ represents the output processed through the argmax function,${\parallel ·\parallel }_1$ denotes the L1 Norm, N is the total number of voxels in the output, $p_{val}^i$ is the predicted value for voxel i, and $g^i$ is the corresponding ground truth. The Jaccard index is also used as an evaluation metric for segmentation performance, as shown in Equation (10).

$$\left\{\begin{array}{c}{\mathcal{P}}_{val}=argmax\left(\mathcal{P}\right),\hfill \\ \parallel {\mathcal{P}}_{val}{\bigcap G\parallel }_1=\sum _{i=1}^N|p_{val}^i\ast g^i|,\hfill \\ DC({\mathcal{P}}_{val},G)=2\frac{\parallel {\mathcal{P}}_{val}{\bigcap G\parallel }_1}{\parallel {\mathcal{P}}_{val}{\parallel }_1+{\parallel G\parallel }_1}.\hfill \end{array}\right.$$

(9)

$$\left\{\begin{array}{c}{\mathcal{P}}_{val}=argmax\left(\mathcal{P}\right),\hfill \\ \parallel {\mathcal{P}}_{val}{\bigcap G\parallel }_1=\sum _{i=1}^N|p_{val}^i\ast g^i|,\hfill \\ \parallel {\mathcal{P}}_{val}{\bigcup G\parallel }_1=\sum _{i=1}^N|p_{val}^i+g^i|,\hfill \\ Jaccard({\mathcal{P}}_{val},G)=\frac{\parallel {\mathcal{P}}_{val}{\bigcap G\parallel }_1}{\parallel {\mathcal{P}}_{val}{\bigcup G\parallel }_1}.\hfill \end{array}\right.$$

(10)

The average symmetric surface distance (ASSD) is used as another evaluation metric to measure the segmentation accuracy of pulmonary lobe boundaries. The formula for ASSD is expressed as in Equation (11). In this equation, v represents any voxel, S represents the set of surface voxels, ${\parallel ·\parallel }_2$ denotes the L2 Norm, $S\left({\mathcal{P}}_{val}\right)$ and $S\left(G\right)$ represent the sets of surface voxels for ${\mathcal{P}}_{val}$ and $\mathit{G}$, respectively, and $\left|S\right|$ indicates the number of elements in a set.

$$\left\{\begin{array}{c}D(v,S)=\underset{s\in S}{min}{\parallel v-s\parallel }_2,\hfill \\ ASSD=\frac{{\sum }_{p\in S\left({\mathcal{P}}_{val}\right)}D(p,S\left(G\right))+{\sum }_{g\in S\left(G\right)}D(g,S\left({\mathcal{P}}_{val}\right))}{|S\left({\mathcal{P}}_{val}\right)|+|S\left(G\right)|}.\hfill \end{array}\right.$$

(11)

The Precision and Recall metrics are also used to measure the model’s performance. The formula for precision and recall is shown in Equations (12) and (13), where $TP$ represents the number of true positives, $FP$ represents the number of false positives, and $FN$ represents the number of false negatives.

$$Precision=\frac{TP}{TP+FP}$$

(12)

$$Recall=\frac{TP}{TP+FN}$$

(13)

4.2. Quantitative Experiment Results

This study conducted experiments on the base dataset using three networks: PLS-Net, RPLS-Net, and nmPLS-Net. The specific experimental results are shown in Table 1a. In this table and the subsequent tables, UR, MR, LR, UL, and LL represent the metric of upper right lobe, middle right lobe, lower right lobe, upper left lobe, and lower left lobe, respectively. The average dice coefficient achieved by nmPLS-Net is 0.9570, the average ASSD is 1.1133, the average precision is 0.9578, the average recall is 0.9615, and the average Jaccard index is 0.9185, surpassing the compared networks. From the experimental results, it can be seen that nmPLS-Net exhibits a significant improvement in overall segmentation performance, especially with a substantial enhancement in the segmentation of the right middle pulmonary lobe. At the same time, nmPLS-Net has also achieved the optimal ASSD value, which means that the network’s segmentation results along the pulmonary lobe edges are more precise. This performance improvement can be attributed to two key factors: Firstly, the nmODE module’s increased memory capacity and high nonlinearity enhance the network’s capability to capture global features and learn dependencies between features. This enables the model to predict results from a more comprehensive and interrelated perspective. Secondly, the fissure-enhanced associative learning approach concentrates the model’s training on voxels near pulmonary fissures and associates them with the fissure information, resulting in an enhanced segmentation performance for challenging fissure voxels. We also trained and validated on another publicly available dataset in [15]. As shown in Table 1b, the results are consistent with those on the base dataset. Our network still achieves the best performance in both overall segmentation and the segmentation of the right middle lobe, demonstrating the effectiveness and reliability of our approach.

Table 1. Comparison of results on different datasets. UR, MR, LR, UL, and LL represent the metric of upper right lobe, middle right lobe, lower right lobe, upper left lobe, and lower left lobe, respectively. The best results in subtables are highlighted with red, and the second best results are highlighted with blue.

(a) Comparison between the target method and previous state-of-art work training on the base dataset.
Model	Metric	RU	RM	RL	LU	LL	AVG
PLS-Net	DC	$0.9309\pm 0.0709$	$0.8690\pm 0.1182$	$0.9511\pm 0.0348$	$0.9442\pm 0.0528$	$0.9441\pm 0.0425$	$0.9279\pm 0.5470$
	ASSD	$3.1171\pm 0.9880$	$1.7678\pm 0.6075$	$1.5129\pm 0.5662$	$2.0548\pm 2.070$	$1.7204\pm 1.2939$	$2.0346\pm 0.7194$
	Precision	$0.9388\pm 0.0410$	$0.8704\pm 0.0896$	$0.9412\pm 0.0502$	$\mathbf{0}.\mathbf{9738}\pm \mathbf{0}.\mathbf{0171}$	$0.9208\pm 0.0912$	$0.9290\pm 0.0221$
	Recall	$0.9555\pm 0.0440$	$0.9078\pm 0.0347$	$0.9173\pm 0.0912$	$\mathbf{0}.\mathbf{9745}\pm \mathbf{0}.\mathbf{0136}$	$0.9404\pm 0.0625$	$0.9391\pm 0.0341$
	Jaccard Index	$0.8717\pm 0.1200$	$0.7683\pm 0.1703$	$0.9058\pm 0.0772$	$0.8943\pm 0.1133$	$0.8943\pm 0.0907$	$0.8668\pm 0.5675$
RPLS-Net	DC	$0.9562\pm 0.0401$	$0.8731\pm 0.1590$	$\mathbf{0.9664}\pm \mathbf{0.0110}$	$0.9601\pm 0.0429$	$0.9546\pm 0.0356$	$0.9421\pm 0.0850$
	ASSD	$1.1829\pm 0.8243$	$2.2384\pm 2.1269$	$\mathbf{0}.\mathbf{8308}\pm \mathbf{0}.\mathbf{2972}$	$1.0739\pm 1.1420$	$1.4468\pm 1.6806$	$1.3546\pm 1.4554$
	Precision	$0.9161\pm 0.0250$	$0.8904\pm 0.0563$	$0.9508\pm 0.0316$	$0.9551\pm 0.0207$	$0.9558\pm 0.0277$	$0.9417\pm 0.0146$
	Recall	$\mathbf{0}.\mathbf{9698}\pm \mathbf{0}.\mathbf{0133}$	$0.9342\pm 0.0249$	$\mathbf{0}.\mathbf{9665}\pm \mathbf{0}.\mathbf{0130}$	$0.9502\pm 0.0836$	$0.9515\pm 0.0197$	$0.9544\pm 0.0152$
	Jaccard Index	$0.9161\pm 0.0423$	$0.7748\pm 0.1876$	$\mathbf{0}.\mathbf{9350}\pm \mathbf{0}.\mathbf{0131}$	$0.9233\pm 0.0542$	$0.9131\pm 0.0411$	$0.8925\pm 0.1246$
nmPLS-Net	DC	$\mathbf{0}.\mathbf{9640}\pm \mathbf{0}.\mathbf{0127}$	$\mathbf{0}.\mathbf{9336}\pm \mathbf{0}.\mathbf{0217}$	$0.9620\pm 0.0198$	$\mathbf{0}.\mathbf{9660}\pm \mathbf{0}.\mathbf{0200}$	$\mathbf{0}.\mathbf{9596}\pm \mathbf{0}.\mathbf{0223}$	$\mathbf{0}.\mathbf{9570}\pm \mathbf{0}.\mathbf{0100}$
	ASSD	$\mathbf{1}.\mathbf{0263}\pm \mathbf{0}.\mathbf{3399}$	$\mathbf{1}.\mathbf{4077}\pm \mathbf{0}.\mathbf{3747}$	$0.9050\pm 0.1429$	$\mathbf{0}.\mathbf{9642}\pm \mathbf{0}.\mathbf{6733}$	$\mathbf{1}.\mathbf{2633}\pm \mathbf{1}.\mathbf{0183}$	$\mathbf{1}.\mathbf{1133}\pm \mathbf{0}.\mathbf{3381}$
	Precision	$\mathbf{0}.\mathbf{9636}\pm \mathbf{0}.\mathbf{0363}$	$\mathbf{0}.\mathbf{9342}\pm \mathbf{0}.\mathbf{0443}$	$\mathbf{0}.\mathbf{9705}\pm \mathbf{0}.\mathbf{0248}$	$0.9632\pm 0.0334$	$\mathbf{0}.\mathbf{9572}\pm \mathbf{0}.\mathbf{0495}$	$\mathbf{0}.\mathbf{9578}\pm \mathbf{0}.\mathbf{0181}$
	Recall	$0.9640\pm 0.0134$	$\mathbf{0}.\mathbf{9583}\pm \mathbf{0}.\mathbf{0188}$	$0.9601\pm 0.0142$	$0.9634\pm 0.0435$	$\mathbf{0}.\mathbf{9618}\pm \mathbf{0}.\mathbf{0183}$	$\mathbf{0}.\mathbf{9615}\pm \mathbf{0}.\mathbf{0095}$
	Jaccard Index	$\mathbf{0}.\mathbf{9300}\pm \mathbf{0}.\mathbf{0319}$	$\mathbf{0}.\mathbf{8781}\pm \mathbf{0}.\mathbf{0337}$	$0.9330\pm 0.029$	$\mathbf{0}.\mathbf{9292}\pm \mathbf{0}.\mathbf{0523}$	$\mathbf{0}.\mathbf{9223}\pm \mathbf{0}.\mathbf{0517}$	$\mathbf{0}.\mathbf{9185}\pm \mathbf{0}.\mathbf{0236}$
(b) Comparison between the target method and previous state-of-art work training on the LUNA16 dataset.
Model	Metric	RU	RM	RL	LU	LL	AVG
PLS-Net	DC	$0.9165\pm 0.0662$	$0.7960\pm 0.1501$	$0.9266\pm 0.0553$	$0.9442\pm 0.0390$	$0.9366\pm 0.0392$	$0.9040\pm 0.0538$
	ASSD	$3.2066\pm 1.3894$	$5.4324\pm 2.5495$	$2.5325\pm 1.2278$	$3.0186\pm 1.858$	$2.7501\pm 0.9402$	$3.3880\pm 1.0641$
	Precision	$\mathbf{0.9326}\pm \mathbf{0.0642}$	$0.7760\pm 0.1971$	$0.9455\pm 0.0219$	$0.9498\pm 0.0385$	$0.9367\pm 0.0455$	$0.9081\pm 0.0538$
	Recall	$0.9185\pm 0.0512$	$0.7847\pm 0.0800$	$\mathbf{0.9500}\pm \mathbf{0.0364}$	$0.9540\pm 0.0161$	$0.9434\pm 0.0051$	$0.9101\pm 0.0152$
	Jaccard Index	$0.8454\pm 0.1120$	$0.6616\pm 0.1799$	$0.8632\pm 0.0694$	$0.8943\pm 0.0703$	$0.8809\pm 0.0753$	$0.8291\pm 0.0985$
RPLS-Net	DC	$\mathbf{0}.\mathbf{9314}\pm \mathbf{0}.\mathbf{0582}$	$0.8291\pm 0.1703$	$0.9407\pm 0.0517$	$0.9548\pm 0.0239$	$0.9465\pm 0.0248$	$0.9205\pm 0.0595$
	ASSD	$2.5011\pm 1.4062$	$3.5855\pm 1.5051$	$1.5885\pm 0.4354$	$1.6182\pm 1.3881$	$1.5146\pm 1.0935$	$2.1616\pm 0.8422$
	Precision	$0.9130\pm 0.0734$	$0.8324\pm 0.1594$	$0.9538\pm 0.0212$	$0.9564\pm 0.0441$	$0.9519\pm 0.0231$	$0.9215\pm 0.0458$
	Recall	$\mathbf{0}.\mathbf{9386}\pm \mathbf{0}.\mathbf{0223}$	$0.8378\pm 0.1479$	$0.9470\pm 0.0355$	$\mathbf{0}.\mathbf{9651}\pm \mathbf{0}.\mathbf{0130}$	$0.9601\pm 0.0171$	$0.9297\pm 0.0358$
	Jaccard Index	$\mathbf{0}.\mathbf{8716}\pm \mathbf{0}.\mathbf{1100}$	$0.7081\pm 0.1944$	$0.8880\pm 0.0899$	$0.9135\pm 0.0694$	$0.8984\pm 0.0657$	$0.8559\pm 0.0899$
nmPLS-Net	DC	$0.9307\pm 0.0550$	$\mathbf{0}.\mathbf{8499}\pm \mathbf{0}.\mathbf{1049}$	$\mathbf{0}.\mathbf{9523}\pm \mathbf{0}.\mathbf{0200}$	$\mathbf{0}.\mathbf{9605}\pm \mathbf{0}.\mathbf{0144}$	$\mathbf{0}.\mathbf{9558}\pm \mathbf{0}.\mathbf{0128}$	$\mathbf{0}.\mathbf{9298}\pm \mathbf{0}.\mathbf{0370}$
	ASSD	$\mathbf{1}.\mathbf{7161}\pm \mathbf{0}.\mathbf{5950}$	$\mathbf{2}.\mathbf{9336}\pm \mathbf{1}.\mathbf{5176}$	$\mathbf{1}.\mathbf{4748}\pm \mathbf{0}.\mathbf{8940}$	$\mathbf{1}.\mathbf{2606}\pm \mathbf{1}.\mathbf{0253}$	$\mathbf{1}.\mathbf{1010}\pm \mathbf{0}.\mathbf{3341}$	$\mathbf{1}.\mathbf{6972}\pm \mathbf{0}.\mathbf{5353}$
	Precision	$0.9011\pm 0.0961$	$\mathbf{0}.\mathbf{8817}\pm \mathbf{0}.\mathbf{0992}$	$\mathbf{0}.\mathbf{9557}\pm \mathbf{0}.\mathbf{0248}$	$\mathbf{0}.\mathbf{9570}\pm \mathbf{0}.\mathbf{0411}$	$\mathbf{0}.\mathbf{9528}\pm \mathbf{0}.\mathbf{0416}$	$\mathbf{0}.\mathbf{9296}\pm \mathbf{0}.\mathbf{0335}$
	Recall	$0.9366\pm 0.0277$	$\mathbf{0}.\mathbf{8876}\pm \mathbf{0}.\mathbf{0642}$	$0.9489\pm 0.0476$	$0.9478\pm 0.0162$	$\mathbf{0}.\mathbf{9739}\pm \mathbf{0}.\mathbf{0057}$	$\mathbf{0}.\mathbf{9390}\pm \mathbf{0}.\mathbf{0341}$
	Jaccard Index	$0.8705\pm 0.1017$	$\mathbf{0}.\mathbf{7390}\pm \mathbf{0}.\mathbf{1403}$	$\mathbf{0}.\mathbf{9089}\pm \mathbf{0}.\mathbf{0716}$	$\mathbf{0}.\mathbf{9241}\pm \mathbf{0}.\mathbf{0553}$	$\mathbf{0}.\mathbf{9151}\pm \mathbf{0}.\mathbf{0527}$	$\mathbf{0}.\mathbf{8715}\pm \mathbf{0}.\mathbf{0775}$

Furthermore, this study conducted training on the training set of the base dataset and subsequently validated the model using 20 randomly selected CT scans from the LUNA16 dataset, as presented in Table 2. Notably, nmPLS-Net continues to exhibit superior performance in the validation results, particularly in the right lung. While the overall performance decreases compared to directly training on the LUNA16 dataset, several factors may contribute to this phenomenon. Firstly, our training dataset comprises only 20 CT scans, in contrast to the 40 CT scans in the training set of the LUNA16 dataset, potentially impacting the model’s generalization ability. Secondly, inconsistencies in data sources and the subjective nature of manual annotations have led to disparities in data distribution between the training and validation sets, contributing to the observed performance drop. However, it is worth highlighting that our model demonstrates resilient performance under these circumstances, especially in segmenting the right middle lobe. This result underscores the enhancing effect of nmPLS-Net’s memory capabilities and the memorized features on the model’s generalization and robustness.

Table 2. Comparison between the target method and previous work training on the base dataset and testing on 20 LUNA16 dataset’s CT volumes. The best results in the table are highlighted with red, and the second best results are highlighted with blue.

Model	Metric	RU	RM	RL	LU	LL	AVG
PLS-Net	DC	$0.8656\pm 0.1198$	$0.7832\pm 0.1688$	$0.9001\pm 0.0731$	$0.9032\pm 0.0808$	$0.8807\pm 0.1097$	$0.8666\pm 0.0833$
	ASSD	$5.1146\pm 3.3692$	$5.7794\pm 2.4364$	$2.8784\pm 0.7087$	$5.0594\pm 4.5588$	$4.4321\pm 3.3269$	$4.6528\pm 1.7337$
	Precision	$0.8693\pm 0.0211$	$0.7258\pm 0.0826$	$0.9464\pm 0.0154$	$0.9313\pm 0.0438$	$0.9059\pm 0.0650$	$0.8757\pm 0.0287$
	Recall	$0.9023\pm 0.0668$	$0.7387\pm 0.0642$	$0.9211\pm 0.0310$	$0.9414\pm 0.0205$	$0.9066\pm 0.0339$	$0.8820\pm 0.0483$
	Jaccard Index	$0.7640\pm 0.1623$	$0.6437\pm 0.1671$	$0.8183\pm 0.1156$	$0.8235\pm 0.1333$	$0.7858\pm 0.1346$	$0.7671\pm 0.1671$
RPLS-Net	DC	$0.8956\pm 0.0532$	$0.8188\pm 0.0679$	$0.9236\pm 0.0286$	$\mathbf{0}.\mathbf{9552}\pm \mathbf{0}.\mathbf{0254}$	$\mathbf{0}.\mathbf{9351}\pm \mathbf{0}.\mathbf{0304}$	$0.9057\pm 0.0358$
	ASSD	$3.6705\pm 1.9807$	$4.1332\pm 1.4911$	$2.8075\pm 1.7102$	$\mathbf{1}.\mathbf{3967}\pm \mathbf{0}.\mathbf{4519}$	$1.9061\pm 0.5417$	$2.7828\pm 1.7246$
	Precision	$0.8788\pm 0.1316$	$0.8198\pm 0.1325$	$0.9519\pm 0.0312$	$\mathbf{0}.\mathbf{9546}\pm \mathbf{0}.\mathbf{0531}$	$0.9462\pm 0.0270$	$0.9102\pm 0.0443$
	Recall	$0.9021\pm 0.0268$	$0.8207\pm 0.1052$	$\mathbf{0}.\mathbf{9546}\pm \mathbf{0}.\mathbf{0398}$	$\mathbf{0}.\mathbf{9462}\pm \mathbf{0}.\mathbf{0298}$	$0.9510\pm 0.0379$	$0.9149\pm 0.0328$
	Jaccard Index	$0.8109\pm 0.1042$	$0.6932\pm 0.1038$	$0.8681\pm 0.0556$	$\mathbf{0}.\mathbf{9131}\pm \mathbf{0}.\mathbf{0622}$	$\mathbf{0}.\mathbf{8782}\pm \mathbf{0}.\mathbf{0867}$	$0.8309\pm 0.1002$
nmPLS-Net	DC	$\mathbf{0}.\mathbf{9197}\pm \mathbf{0}.\mathbf{0438}$	$\mathbf{0}.\mathbf{8508}\pm \mathbf{0}.\mathbf{0539}$	$\mathbf{0}.\mathbf{9381}\pm \mathbf{0}.\mathbf{0227}$	$0.9299\pm 0.0382$	$0.9238\pm 0.0281$	$\mathbf{0}.\mathbf{9125}\pm \mathbf{0}.\mathbf{0303}$
	ASSD	$\mathbf{2}.\mathbf{2433}\pm \mathbf{1}.\mathbf{7509}$	$\mathbf{3}.\mathbf{5638}\pm \mathbf{2}.\mathbf{2124}$	$\mathbf{1}.\mathbf{4560}\pm \mathbf{0}.\mathbf{4983}$	$1.5622\pm 1.3761$	$\mathbf{1}.\mathbf{3036}\pm \mathbf{0}.\mathbf{3838}$	$\mathbf{2}.\mathbf{0258}\pm \mathbf{0}.\mathbf{7938}$
	Precision	$\mathbf{0}.\mathbf{8853}\pm \mathbf{0}.\mathbf{0627}$	$\mathbf{0}.\mathbf{8711}\pm \mathbf{0}.\mathbf{0895}$	$\mathbf{0}.\mathbf{9532}\pm \mathbf{0}.\mathbf{0155}$	$0.9447\pm 0.0247$	$\mathbf{0}.\mathbf{9592}\pm \mathbf{0}.\mathbf{0275}$	$\mathbf{0}.\mathbf{9227}\pm \mathbf{0}.\mathbf{0184}$
	Recall	$\mathbf{0}.\mathbf{9150}\pm \mathbf{0}.\mathbf{0583}$	$\mathbf{0}.\mathbf{8831}\pm \mathbf{0}.\mathbf{0656}$	$0.9484\pm 0.0463$	$0.9368\pm 0.0223$	$\mathbf{0}.\mathbf{9696}\pm \mathbf{0}.\mathbf{0108}$	$\mathbf{0}.\mathbf{9306}\pm \mathbf{0}.\mathbf{0157}$
	Jaccard Index	$\mathbf{0}.\mathbf{8513}\pm \mathbf{0}.\mathbf{0908}$	$\mathbf{0}.\mathbf{7403}\pm \mathbf{0}.\mathbf{1045}$	$\mathbf{0}.\mathbf{8834}\pm \mathbf{0}.\mathbf{0512}$	$0.8690\pm 0.0499$	$0.8584\pm 0.0697$	$\mathbf{0}.\mathbf{8405}\pm \mathbf{0}.\mathbf{0817}$

4.3. Ablation Experiment

Ablation experiments were conducted to analyze the contributions of various components in the methodology for performance improvement. The results of the ablation experiments are shown in Table 3. In cases where the nmODE module and fissure-enhanced associative learning were not used, the model reverted to PLS-Net, so separate experiments for this scenario were not conducted again. It can be observed that both the nmODE module and fissure-enhanced associative learning contribute to the segmentation performance of the model. The nmODE module, in particular, exhibits the most significant improvement in segmentation performance, with the performance boost primarily evident in the right middle lobe.

Table 3. Ablation experiments, trained and tested on the base dataset. The right symbols at the first two columns represent the use of the corresponding module. FEAL represents fissure-enhanced associative learning. The best results in the table are highlighted with red, and the second best results are highlighted with blue.

nmODE	FEAL	Metric	RU	RM	RL	LU	LL	AVG
		DC	$0.9309\pm 0.0709$	$0.8690\pm 0.1182$	$0.9511\pm 0.0348$	$0.9442\pm 0.0528$	$0.9441\pm 0.0425$	$0.9279\pm 0.5470$
		ASSD	$1.7678\pm 0.6075$	$3.1171\pm 0.988$	$1.5129\pm 0.5662$	$2.0548\pm 2.0700$	$1.7204\pm 1.2939$	$2.0346\pm 0.7194$
		Precision	$0.9388\pm 0.0410$	$0.8704\pm 0.0896$	$0.9412\pm 0.0502$	$\mathbf{0}.\mathbf{9738}\pm \mathbf{0}.\mathbf{0171}$	$0.9208\pm 0.0912$	$0.9290\pm 0.0221$
		Recall	$0.9555\pm 0.0440$	$0.9078\pm 0.0347$	$0.9173\pm 0.0912$	$\mathbf{0}.\mathbf{9745}\pm \mathbf{0}.\mathbf{0136}$	$0.9404\pm 0.0625$	$0.9391\pm 0.0341$
		Jaccard Index	$0.8717\pm 0.1200$	$0.7683\pm 0.1703$	$0.9058\pm 0.0772$	$0.8943\pm 0.1133$	$0.8943\pm 0.0907$	$0.8668\pm 0.5675$
		DC	$0.9648\pm 0.0066$	$0.9187\pm 0.0457$	$0.9655\pm 0.0087$	$0.9553\pm 0.0188$	$0.9446\pm 0.0458$	$0.9498\pm 0.0190$
		ASSD	$1.2216\pm 0.5127$	$1.8847\pm 0.6801$	$1.0419\pm 0.1517$	$\mathbf{0}.\mathbf{9042}\pm \mathbf{0}.\mathbf{3784}$	$\mathbf{1}.\mathbf{0596}\pm \mathbf{0}.\mathbf{4025}$	$1.2224\pm 0.2559$
	✓	Precision	$0.9600\pm 0.0362$	$0.9200\pm 0.034$	$0.9653\pm 0.0105$	$0.9507\pm 0.0367$	$0.9504\pm 0.0486$	$0.9493\pm 0.0183$
		Recall	$0.9590\pm 0.0061$	$0.9372\pm 0.0219$	$0.9597\pm 0.0116$	$0.9449\pm 0.0921$	$0.9509\pm 0.0130$	$0.9523\pm 0.0180$
		Jaccard Index	$0.9320\pm 0.0185$	$0.8496\pm 0.1012$	$0.9333\pm 0.0636$	$0.9144\pm 0.0692$	$0.8950\pm 0.1024$	$0.9049\pm 0.0675$
		DC	$\mathbf{0}.\mathbf{9692}\pm \mathbf{0}.\mathbf{0075}$	$0.9250\pm 0.0331$	$\mathbf{0}.\mathbf{9660}\pm \mathbf{0}.\mathbf{0062}$	$\mathbf{0}.\mathbf{9676}\pm \mathbf{0}.\mathbf{0234}$	$0.9556\pm 0.0237$	$0.9567\pm 0.0110$
		ASSD	$\mathbf{0}.\mathbf{9574}\pm \mathbf{0}.\mathbf{2561}$	$1.5769\pm 0.4282$	$0.9529\pm 0.1772$	$1.0195\pm 0.7127$	$1.2403\pm 0.8388$	$1.1494\pm 0.3035$
✓		Precision	$0.9635\pm 0.0420$	$0.9276\pm 0.0843$	$0.9679\pm 0.0152$	$0.9705\pm 0.0178$	$0.9457\pm 0.0206$	$0.9550\pm 0.0200$
		Recall	$\mathbf{0}.\mathbf{9659}\pm \mathbf{0}.\mathbf{0068}$	$0.9488\pm 0.0230$	$\mathbf{0}.\mathbf{9628}\pm \mathbf{0}.\mathbf{0149}$	$0.9633\pm 0.0316$	$0.9576\pm 0.0182$	$0.9597\pm 0.0077$
		Jaccard Index	$\mathbf{0}.\mathbf{9338}\pm \mathbf{0}.\mathbf{0122}$	$0.8603\pm 0.0425$	$\mathbf{0}.\mathbf{9397}\pm \mathbf{0}.\mathbf{0149}$	$\mathbf{0}.\mathbf{9385}\pm \mathbf{0}.\mathbf{0410}$	$0.9182\pm 0.0472$	$0.9181\pm 0.0166$
		DC	$0.9640\pm 0.0127$	$\mathbf{0}.\mathbf{9336}\pm \mathbf{0}.\mathbf{0217}$	$0.9620\pm 0.0198$	$0.9660\pm 0.0200$	$\mathbf{0}.\mathbf{9596}\pm \mathbf{0}.\mathbf{0223}$	$\mathbf{0}.\mathbf{9570}\pm \mathbf{0}.\mathbf{0100}$
		ASSD	$1.0263\pm 0.3399$	$\mathbf{1}.\mathbf{4077}\pm \mathbf{0}.\mathbf{3747}$	$\mathbf{0}.\mathbf{9050}\pm \mathbf{0}.\mathbf{1429}$	$0.9642\pm 0.6733$	$1.2633\pm 1.0183$	$\mathbf{1}.\mathbf{1133}\pm \mathbf{0}.\mathbf{3381}$
✓	✓	Precision	$\mathbf{0}.\mathbf{9636}\pm \mathbf{0}.\mathbf{0363}$	$\mathbf{0}.\mathbf{9342}\pm \mathbf{0}.\mathbf{0443}$	$\mathbf{0}.\mathbf{9705}\pm \mathbf{0}.\mathbf{0248}$	$0.9632\pm 0.0334$	$\mathbf{0}.\mathbf{9572}\pm \mathbf{0}.\mathbf{0495}$	$\mathbf{0}.\mathbf{9578}\pm \mathbf{0}.\mathbf{0181}$
		Recall	$0.9641\pm 0.0134$	$\mathbf{0}.\mathbf{9583}\pm \mathbf{0}.\mathbf{0188}$	$0.9601\pm 0.0142$	$\mathbf{0}.\mathbf{9634}\pm \mathbf{0}.\mathbf{0435}$	$\mathbf{0}.\mathbf{9618}\pm \mathbf{0}.\mathbf{0183}$	$\mathbf{0}.\mathbf{9615}\pm \mathbf{0}.\mathbf{0095}$
		Jaccard Index	$0.9300\pm 0.0319$	$\mathbf{0}.\mathbf{8781}\pm \mathbf{0}.\mathbf{0337}$	$0.9330\pm 0.029$	$0.9292\pm 0.0523$	$\mathbf{0}.\mathbf{9223}\pm \mathbf{0}.\mathbf{0517}$	$\mathbf{0}.\mathbf{9185}\pm \mathbf{0}.\mathbf{0236}$

Although adding fissure-enhanced associative learning on top of the nmODE module does not substantially improve overall segmentation performance, the enhancement in the segmentation performance of the right middle lobe is notable, indicating that fissure-enhanced associative learning does have an effect, aligning with the purpose of their incorporation into our methodology.

The results of the ablation experiments on the loss functions are presented in Table 4. Only the mean values of each metric are listed to simplify the table. The experimental results indicate that using a mixed loss function for pulmonary lobe segmentation tasks can lead to some improvement, although it has minimal overall impact. It can also be observed that dice or cross-entropy, as the loss function for the FEAL module, is less effective than not using the FEAL module. This is primarily due to the extreme class imbalance in the labels used by FEAL, where both dice and cross-entropy struggle to effectively calculate the loss contributions of the sparsely represented positive samples.

Table 4. Ablation experiments for the loss function, trained and tested on the base dataset. mDC, mASSD, mPrecision, mRecall and mJaccard-Index are the mean dice coefficient (DC), mean average symmetric surface distance (ASSD), mean precision, mean recall, and mean Jaccard index, respectively. The content within the brackets in the table header represents (${\mathcal{L}}_{main}$, ${\mathcal{L}}_e$). The best results in the table are highlighted with red, and the second best results are highlighted with blue.

Metric	(CE, Focal)	(Dice, Focal)	(CE+Dice, Focal)	(Dice+Focal, Dice)	(Dice+Focal, CE)	(CE+Focal, Focal)	(Dice+Focal, Focal)
mDC	$0.9519\pm 0.0192$	$0.9552\pm 0.0238$	$0.9563\pm 0.0145$	$0.9531\pm 0.0178$	$0.9540\pm 0.0178$	$0.9558\pm 0.0126$	$\mathbf{0}.\mathbf{9570}\pm \mathbf{0}.\mathbf{0100}$
mASSD	$1.2538\pm 0.4015$	$1.5171\pm 0.5332$	$1.1427\pm 0.3955$	$1.1411\pm 0.4172$	$1.2511\pm 0.4318$	$1.1503\pm 0.3524$	$\mathbf{1}.\mathbf{1133}\pm \mathbf{0}.\mathbf{3381}$
mPrecision	$0.9528\pm 0.0190$	$0.9517\pm 0.0336$	$0.9556\pm 0.0202$	$0.9509\pm 0.0259$	$0.9538\pm 0.0271$	$0.9566\pm 0.0252$	$\mathbf{0}.\mathbf{9578}\pm \mathbf{0}.\mathbf{0181}$
mRecall	$0.9581\pm 0.0099$	$0.9545\pm 0.0162$	$0.9597\pm 0.0131$	$0.9540\pm 0.0209$	$0.9578\pm 0.0125$	$0.9606\pm 0.0113$	$\mathbf{0}.\mathbf{9615}\pm \mathbf{0}.\mathbf{0095}$
mJaccard-Index	$0.9092\pm 0.0343$	$0.9141\pm 0.0544$	$0.9159\pm 0.0485$	$0.9120\pm 0.0415$	$0.9131\pm 0.0373$	$0.9153\pm 0.0447$	$\mathbf{0}.\mathbf{9186}\pm \mathbf{0}.\mathbf{0236}$

4.4. Qualitative Analysis

Quantitative evaluation validates the effectiveness of nmPLS-Net in segmenting right pulmonary lobes and in case of unclear pulmonary fissures. Figure 4 showcases the segmentation outcomes achieved by our network and the comparative networks across five distinct CT scans. In the five CT cases, the first CT represents a normal lung with intact and clear pulmonary fissures. In the remaining four CT cases, partial or complete unclear pulmonary fissures are present. It can be observed that in the CTs of rows 2, 3, and 5, nmPLS-Net consistently demonstrates superior performance. In contrast, the other networks exhibit poor performance, with some results displaying noticeable segmentation errors. While the segmentation results in the CT scans of the third row are generally suboptimal, nmPLS-Net still provides the closest approximation to the ground truth. This case underscores nmPLS-Net’s adeptness at effectively handling scenarios with unclear or incomplete pulmonary fissures.

Figure 4. The segmentation results of three CT scans in nmPLS-Net and the comparative networks. In this visual representation, distinct lung regions are color-coded: yellow for the right upper lobe, green for the right middle lobe, brown for the right lower lobe, blue for the left upper lobe, and red for the left lower lobe. The first column represents the original images, and the subsequent four columns represent the segmentation results for the ground truth, PLS-Net, RPLS-Net, and nmPLS-Net, respectively.

To further substantiate the efficacy of nmODE, an analysis was conducted on feature maps generated by the model using six CT scans. These feature maps included those with dimensions $(129,\mathit{D}/8,\mathit{H}/8,\mathit{W}/8)$ before entering the nmODE module, as well as those with the same dimensions after passing through the nmODE module. Principal component analysis (PCA) was applied to reduce the dimensionality of these feature maps from $(129,\mathit{D}/8,\mathit{H}/8,\mathit{W}/8)$ to a compact representation of $(129,2)$. These reduced-dimensional feature representations were then visualized on a 2D plane, and the outcomes are presented in Figure 5. Figure 5a provides a visual representation of features before entering the nmODE module, while Figure 5b showcases the features after undergoing processing by the nmODE module. Each color in the visualizations corresponds to the same CT scan across both figures. It can be observed that the original features, after being processed by nmODE, exhibit more pronounced overall similarity in shape, indicating that nmODE’s memory capacity can indeed extract common features among different samples. Furthermore, the distribution of features before processing by the nmODE module is more concentrated. In contrast, the feature distribution following nmODE processing exhibits a heightened level of uniformity and reveals stronger inter-feature correlations. These findings underscore the nmODE module’s adeptness in learning intricate feature correlations, thus enhancing its capacity to capture meaningful relationships within the data.

Figure 5. Visualized feature-maps of 6 CT volumes. In (a,b), the same colors represent the output features of the same CT volumes and each scatterplot in the figure is obtained by projecting the various channels of the model’s encoder output features onto a two-dimensional plane after PCA dimension reduction. (a) shows the scatterplot of the feature mapping obtained before processing with the nmODE module, while (b) shows the scatterplot of the feature mapping obtained after processing with the nmODE module.

5. Discussion and Conclusions

This study proposes a 3D fully convolutional neural network, nmPLS-Net, which combines nmODE (nonlinear memory ordinary differential equation) with multitask learning to perform the end-to-end segmentation of pulmonary lobes in chest CT scans. The memory capabilities and high nonlinearity of the nmODE module enhance the extraction of global features, improving the model’s performance in segmenting the overall lung and complex scenarios in the right lung. Additionally, fissure-enhanced associative learning guides the model’s attention toward challenging voxels within pulmonary fissures. The model’s performance is validated on two distinct datasets, and the experimental results demonstrate that nmPLS-Net achieves notably superior results in right pulmonary lobe segmentation compared to previous methods. However, incorporating nmODE requires an ODE solver, which may slightly reduce computational speed. This problem can be addressed in future work through knowledge distillation, an area we plan to explore for further improvements.