SALM: A Unified Model for 2D and 3D Region of Interest Segmentation in Lung CT Scans Using Vision Transformers

Abstract

Accurate segmentation of Regions of Interest (ROI) in lung Computed Tomography (CT) is crucial for early lung cancer diagnosis and treatment planning. However, the variability in size, shape, and location of lung lesions, along with the complexity of 3D spatial relationships, poses significant challenges. In this work, we propose SALM (Segment Anything in Lung Model), a deep learning model for 2D and 3D ROI segmentation. SALM leverages Vision Transformers, proposing an adaptation of positional encoding functions to effectively capture spatial relationships in both 2D slices and 3D volumes using a single, unified model. Evaluation on the LUNA16 dataset demonstrated strong performance in both modalities. In 2D segmentation, SALM achieved a Dice score of 93% on 124,662 slices. For 3D segmentation using 174 3D images from the same dataset, SALM attained a Dice score of 81.88%. We also tested SALM on an external database (PleThora) on a subset of 255 pulmonary CT from diseased patients, where it achieved a Dice score of 78.82%. These results highlight SALM’s ability to accurately segment lung ROI in both 2D and 3D, demonstrating its potential to improve the accuracy and efficiency of computer-aided diagnosis for lung cancer.

1. Introduction

Lung cancer remains a leading cause of cancer-related mortality worldwide [1]. Early detection and accurate diagnosis are crucial for improving patient outcomes. CT imaging plays a vital role in the diagnosis and treatment planning of lung cancer, providing detailed cross-sectional images of the lungs. Within these images, the precise segmentation of ROI, such as lung lesions and surrounding anatomical structures, is essential for accurate disease assessment, staging, and treatment planning [2]. However, manual segmentation of ROI in lung CT images is a time-consuming and labor-intensive process, prone to inter-observer variability.

The development of automated and reliable methods for lung CT image segmentation has thus become a critical area of research. While traditional image processing techniques have been applied, they often face limitations in handling the inherent complexities of lung CT images [3]. These complexities include variations in lesion size, shape, density, and location, as well as the presence of similar-appearing structures like blood vessels. Furthermore, segmenting 3D CT volumes adds another layer of complexity due to the need to capture complex spatial relationships across multiple slices.

Recently, deep learning technologies have emerged as powerful tools for medical image analysis, offering promising solutions for automated lung CT segmentation [4,5,6]. Convolutional Neural Networks (CNNs) have demonstrated exceptional performance in various image recognition tasks, including medical image segmentation [7,8]. However, CNNs are limited in their ability to capture long-range dependencies, which are crucial for understanding the global context within medical images [9].

The emergence of Transformers [10], originally developed for natural language processing, has revolutionized the field of computer vision. Transformers leverage self-attention mechanisms to model relationships between different parts of an input sequence, enabling them to effectively capture long-range dependencies. This capability makes them particularly well suited for analyzing complex medical images, where understanding the spatial relationships between different anatomical structures is essential.

Furthermore, the use of Foundation Models (FMs) [11], which are large-scale models pretrained on massive datasets, has further enhanced the performance of Vision Transformers. These FMs learn rich visual representations from diverse data, providing a strong starting point for fine-tuning on specific tasks, such as lung CT segmentation.

In this study, we present SALM (Segment Anything in Lung Model), a deep learning model for both 2D and 3D ROI segmentation in lung CT images. SALM is built upon the Vision Transformer architecture and incorporates a unique adaptation of positional encoding using weights of the FM Segment Anything Model (SAM) from Meta AI [12]. This adaptation, inspired by the need to capture spatial relationships at multiple scales within lung CT volumes, involves modifying the periodicity of the standard sinusoidal functions used in positional encoding. This allows SALM to effectively integrate information across different spatial scales, improving its ability to delineate ROI accurately. Another major contribution of our work lies in our approach to managing the size variability of lung CT images. Our SALM incorporates advanced image processing techniques to standardize image scale and resolution during preprocessing, enabling the accurate segmentation of CT images, irrespective of their original size. This standardization allows the model to maintain high accuracy even on smaller original images by ensuring a standardized and optimal size input for the network.

Our model is refined by incorporating elements from MedSAM [13], specifically tailored for medical image segmentation. We leverage a unified architecture capable of processing both 2D slices and 3D volumes without requiring separate training, making SALM a versatile tool for comprehensive lung image analysis.

We evaluate SALM’s performance on the publicly available LUNA16 dataset [14], a benchmark dataset for lung nodule analysis. Our experiments demonstrate that SALM achieves strong performance in both 2D and 3D segmentation tasks. In 2D segmentation, SALM achieves an Area under the Curve (AUC) of 99% and a Dice Similarity Coefficient (DSC) of 93% on 124,662 slices. In 3D segmentation, evaluated on 174 3D images, SALM attains an F1-score of 75.57% and a Dice score of 81.88%. We also tested SALM on a subsample of 255 pulmonary images from diseased individuals’ contents in the PleThora dataset [15,16], where it achieved a DICE performance of 78.82% and an accuracy of 97.64%.These results underline SALM’s ability to accurately segment ROI in lung CT images, highlighting its potential to enhance the accuracy and efficiency of computer-aided diagnosis for lung cancer.

The main contributions of this work are as follows:

• Development of a Unified 2D and 3D Segmentation Model: We propose SALM, a unified Vision Transformer-based model capable of performing both 2D and 3D ROI segmentation in lung CT images. This unified architecture simplifies the segmentation process and demonstrates the versatility of Vision Transformers in handling different data modalities within a single framework.
• Novel Adaptation of Positional Encoding for Enhanced Spatial Context Awareness: We demonstrate that modifying the periodicity of the standard sinusoidal functions used in positional encoding significantly enhances the model’s ability to capture multi-scale spatial relationships, particularly within 3D volumetric data. By introducing a $3\pi$ modulation factor, we effectively increase the spatial frequency of the encoding. This allows the model to integrate information across different scales, leading to improved segmentation accuracy, especially in complex structures like those found in lung CT images. This finding provides insights into optimizing Transformer architectures for volumetric medical image analysis.
• Adaptation to Lung Image Size Variability: Another major contribution of our work lies in our approach to managing the size variability of lung CT images. Our SALM model incorporates advanced image processing techniques to standardize image scale and resolution during preprocessing, enabling accurate segmentation of CT images, irrespective of their original size. This standardization not only ensures high accuracy in the detection and segmentation of ROI of varying sizes, but also helps maintain optimal model performance on a wide range of lung imaging data. By directly tackling the challenge posed by the diversity of lung CT images, we are helping to make lung image segmentation models efficient and reliable, without compromising performance even for smaller original images.

The remainder of this paper is organized as follows: Section 2 provides an overview of related work in medical image segmentation, with a focus on lung CT analysis. Section 3 details the proposed SALM model, including the architecture, the modified positional encoding, and the training methodology. Section 4 presents the experimental results, both quantitative and qualitative, and discusses the performance of SALM. Finally, Section 5 concludes the paper and outlines directions for future research. Additionally, Appendix A provides a detailed analysis of the carbon footprint associated with the development and training of the SALM model, reflecting our commitment to responsible and sustainable AI research.

2. Related Works

The field of medical image segmentation has witnessed significant advancements, driven by the increasing availability of medical imaging data and the development of powerful computational techniques [17]. In particular, the segmentation of lung CT images has received considerable attention due to its crucial role in the diagnosis, treatment planning, and monitoring of lung diseases, especially lung cancer.

Early approaches to lung CT segmentation relied on traditional image processing techniques, such as thresholding, region growing, and active contour models [18]. While these methods can achieve reasonable results in simple cases, they demonstrate limitations in handling the complex and heterogeneous nature of lung pathologies, as well as variations in image quality and acquisition parameters. For instance, Bellotti et al. [18] proposed a system based on region growing and active contour models for lung nodule detection, achieving a detection rate of 88.5% with 6.6 false positives per CT scan on a limited dataset.

More recently, supervised methods, such as the one proposed by Li et al. [19], have utilized geometric active contours for semi-3D segmentation of lung tissue. These methods have shown improvements in accuracy and speed, particularly for segmenting pleural nodules. However, they often require manual initialization or interaction, which can limit their applicability in large-scale studies.

The introduction of deep learning has revolutionized the field of medical image segmentation, with CNN demonstrating significant success in a wide range of applications [17,20,21]. In the context of lung CT segmentation, CNN-based methods have been extensively explored for tasks such as lung field segmentation, nodule detection, and tumor segmentation. Wu et al. [22] proposed a 3D-UNET optimized by a three-dimensional conditional random field (3D-CRF) for pulmonary nodule segmentation, achieving a Dice score of 80.1% on the LIDC-IDRI dataset. Zhao et al. [23] developed a deep learning system based on 3D CNN and multitask learning for lung nodule classification and segmentation, outperforming radiologists in terms of F1-score.

Several studies have focused on specific challenges in lung CT segmentation, such as handling variations in nodule size and shape. For example, Wang et al. [24] developed a 3D CNN with a stratified training strategy for automated lung tumor segmentation in radiotherapy planning, achieving a Dice Similarity Coefficient of 83% for large tumor volumes. Riaz et al. [25] addressed the segmentation of cancerous lesions by merging MobileNetV2 and UNET architectures, demonstrating improved performance on the Medical Segmentation Decathlon (MSD) dataset.

Despite their success, CNN-based methods have inherent limitations in capturing long-range dependencies within images, which can be crucial for accurately segmenting complex structures in lung CT scans. This limitation has motivated the exploration of alternative architectures, such as Transformers [10], which have shown great promise in natural language processing and, more recently, in computer vision tasks [26].

Transformers leverage self-attention mechanisms to model relationships between different parts of an input sequence, enabling them to effectively capture global context. This capability has led to their increasing adoption in medical image analysis, including lung CT segmentation. Several recent studies have explored the use of Transformers for lung nodule detection, classification, and segmentation [27,28,29,30]. However, the application of Transformers to 3D medical image segmentation, particularly for lung CT scans, remains an active area of research.

Our work builds upon these recent advances and proposes a Transformer-based model, SALM, for both 2D and 3D ROI segmentation in lung CT images. SALM leverages a unique adaptation of positional encoding, specifically designed to capture multi-scale spatial relationships within lung CT volumes. This adaptation, which involves modifying the periodicity of the standard sinusoidal functions used in positional encoding, enables SALM to effectively integrate information across different spatial scales. Furthermore, SALM utilizes a unified architecture capable of processing both 2D slices and 3D volumes, making it a versatile tool for comprehensive lung image analysis.

Table 1. Summary of work on lung CT segmentation.

Ref.	Objective	Database	Results (%)
[31]	Detection of pulmonary nodules	LUNA16 [14]	DSC = 88.89
[32]	Early detection of lung cancer	MSD [33]	Seg. accuracy = 97.83
			Class. accuracy = 98.77
[24]	Tumor segmentation for radiotherapy planning	Private clinical database	DSC = 83
[25]	Cancerous lesion segmentation	MSD [33]	DSC = 87.93
			Recall = 86.02
			Precision = 93
[34]	Accurate tumor segmentation in NSCLC treatment	NSCLC cases	DSC = 80
[35]	Segmentation	134 scans CT	Dice = 88.8
			Sensitivity = 87.2
			Precision = 90.9
[36]	Recognition of lung cancer	LIDC-IDRI [37]	Sensitivity = 95.7
[38]	Lung cancer detection challenge	ACDC@LungHP Database [39]	-
[40]	Lung cancer segmentation	NSCLC	DSC = 86.1
[41]	Automatic tumor segmentation from T2 MRI	377 patients CT + 6 IRM	DSC = 80
[18]	Automatic detection of lung nodules using region growth algorithms and active contour models	15 CT scans with approx. 4700 slices	Detection rate: 88.5, false positives: 6.6 per CT
[19]	Supervised semi-3D segmentation of lung tissue and reconstruction of untrimmed 3D models	15 scans of healthy and early-stage lung tumor patients	Outperforms in accuracy and speed
[22]	Accurate segmentation of pulmonary nodules using 3D-UNET network optimized by 3D-CRF	936 lung nodules from LIDC-IDRI, validated on clinical data.	Dice score: 80.1
[23]	Automatic classification and segmentation of lung nodules using 3D CNN and multitask learning	651 nodules annotated with segmentation masks and pathological labels	Weighted average F1-score: 63.3 vs. radiologists 51.0 to 56.6

Seg: segmentation; Class: classification.

summarizes these related works, presenting the objectives, databases used, and results obtained.

3. Methods

In this section, we detail the proposed SALM model for 2D and 3D ROI segmentation in lung CT images. We describe the dataset used; the preprocessing steps; the model architecture, with a particular emphasis on the novel adaptation of positional encoding; and the training procedure.

3.1. Dataset and Preprocessing

The LUNA16 (Lung Nodule Analysis 2016) dataset [14] was used for training and evaluating our model. This publicly available dataset is a subset of the larger LIDC-IDRI database and contains 888 thoracic CT scans with accompanying annotations for lung nodules. The scans are provided in the MetaImage format (.mhd/.raw) and have a typical resolution of 512 × 512 pixels per slice (Figure 1), with varying slice thicknesses ranging from 0.6 to 3 mm.

To prepare the data for our model, we performed several preprocessing steps. First, since LUNA16 focuses on lung imaging, we normalized the Hounsfield units (HU) to standardized window width and level values (W: 1500, L: − 160) specifically for the lung, as recommended in [42]. This important step homogenizes pixel intensity values and highlights the anatomical structures of interest in the lung. Following this, the intensity values were scaled to a range of [0, 255].

In order to standardize the image size and resolution and ensure consistent input for the model, we applied the following technical steps:

• Isotropic resampling (for 3D images): We first resampled each CT volume to an isotropic voxel spacing of 1 × 1 × 1 mm using linear interpolation. This step, performed using linear interpolation functions available in standard imaging libraries, ensures that spatial relationships between voxels are consistent across all dimensions, regardless of the original slice spacing.
• Slice-by-slice processing: Each 3D volume was then processed slice by slice. This approach allows us to apply SALM, designed for 2D inputs, to 3D volumetric data. While the Vision Transformer processes each slice independently, the modified 3D positional encoding, described in detail in Section 3.2.2, allows for the integration of 3D spatial context.
• Resizing to 1024 × 1024: Each slice, whether from an original 2D image or extracted from a 3D volume, was resized to a standard size of 1024 × 1024 pixels using linear interpolation (linear interpolation). This resizing operation ensures that the model receives input images of uniform dimensions, which is crucial for the performance of the Vision Transformer. The choice of the 1024 × 1024 size represents a compromise between preserving anatomical details and memory and computational constraints.
• Channel repetition (for 2D images): For original 2D images and slices extracted from 3D volumes, we repeated the intensity channel three times to create a 3-channel input image, compatible with the ViT architecture and consistent with the processing of color images, even though CT images are grayscale.

While each slice is processed as a 2D image by the Vision Transformer, the 3D positional encoding provides crucial context about the slice’s location within the entire 3D volume. This combined approach allows SALM to efficiently process 3D data while leveraging the power of 2D Vision Transformers.

We then resampled each CT volume to an isotropic voxel spacing of 1 × 1 × 1 mm using linear interpolation. This step ensures that the spatial relationships between voxels are consistent across all dimensions. Subsequently, each 3D volume was processed slice by slice, with each slice resized to 1024 × 1024 pixels and the channel repeated three times, as in the 2D case. This approach allows us to leverage the same model architecture for both 2D and 3D segmentation while explicitly incorporating 3D spatial information through our modified positional encoding, which is described in detail below. While each slice is processed as a 2D image by the Vision Transformer, the 3D positional encoding provides crucial context about the slice’s location within the entire 3D volume.

3.2. Model Architecture

SALM is based on the Vision Transformer (ViT) architecture [26], which has demonstrated remarkable performance in various computer vision tasks. We adopt the base ViT model (ViT-B) due to its balance between computational efficiency and accuracy. Larger ViT variants (e.g., ViT-L, ViT-H) offer marginal improvements in accuracy at the cost of significantly increased computational requirements [43,44], making them less suitable for our task, where processing large 3D volumes is essential.

The SALM architecture consists of three main components: an image encoder, a 3D positional encoder, and a mask decoder. Figure 2 illustrates the overall architecture of SALM.

3.2.1. Image Encoder

The image encoder is responsible for extracting visual features from the input CT slices. It follows the standard ViT architecture, which processes the input image as a sequence of patches. Given an input slice of size 1024 × 1024 × 3, we divide it into a sequence of non-overlapping patches of size 16 × 16 × 3. Each patch is then linearly projected into a fixed-dimensional embedding vector. These patch embeddings, along with the positional encodings, are fed into a series of Transformer layers. Each Transformer layer consists of a Multi-head Self-Attention (MSA) module and a Feedforward Network (FFN), with layer normalization applied before each module and residual connections applied after each module. The image encoder produces a feature map that captures both local and global visual information from the input slice (Figure 3).

3.2.2. 3D Positional Encoding

To enable SALM to effectively process 3D volumes, we propose a novel adaptation of the standard positional encoding used in Transformers. While the image encoder processes each slice independently, the 3D positional encoding provides crucial context about the slice’s location within the 3D volume, allowing the model to learn spatial relationships across slices.

Standard positional encodings in Transformers use sinusoidal functions to generate unique positional vectors for each element in a sequence. These encodings are typically defined as

$$PE(pos,2i)=sin\left({\displaystyle \frac{pos}{{10,000}^{2i/d}}}\right)$$

(1)

$$PE(pos,2i+1)=cos\left({\displaystyle \frac{pos}{{10,000}^{2i/d}}}\right)$$

(2)

where $pos$ is the position of the element in the sequence, i is the dimension index, and d is the dimensionality of the embedding.

In our adaptation, we modify the periodicity of these functions by introducing a factor of $3\pi$, as follows:

$$PE(pos,2i)=sin\left({\displaystyle \frac{pos·3\pi }{{10,000}^{2i/d}}}\right)$$

(3)

$$PE(pos,2i+1)=cos\left({\displaystyle \frac{pos·3\pi }{{10,000}^{2i/d}}}\right)$$

(4)

This modification effectively increases the spatial frequency of the encoding, allowing it to capture finer-grained positional information (Figure 4). The $3\pi$ factor was empirically determined to be optimal for our task through experimentation with different values, which showed that it provided the best balance between capturing detailed spatial relationships and avoiding overfitting to specific positional patterns.

The choice of the $3\pi$ factor to modulate the periodicity of the sinusoidal functions in positional encoding is based on a primarily empirical approach. Through a series of experiments we observed that this value allowed us to achieve the best segmentation performance on the LUNA16 dataset. The increase in the spatial frequency of the encoding could promote the capture of finer details and multi-scale spatial relationships specific to the complex structures of lung CT images.

For each 3D volume, we generate a unique 3D positional encoding for each slice based on its spatial coordinates (x, y, z) within the volume. The x and y coordinates correspond to the patch position within the slice, while the z coordinate represents the slice index. These 3D positional encodings are added to the corresponding patch embeddings before being fed into the Transformer layers, providing the model with explicit information about the spatial location of each patch within the 3D volume.

3.2.3. Mask Decoder

The mask decoder is responsible for generating the final segmentation mask from the encoded image features and positional information. It consists of a series of Transformer decoder layers, followed by a simple upsampling module. Each Transformer decoder layer takes as input the output of the image encoder and the positional encodings, and uses multi-head self-attention and cross-attention mechanisms to integrate information from both the image features and the positional context.

The upsampling module progressively increases the spatial resolution of the feature maps using transposed convolutions until it matches the original input resolution (1024 × 1024). Finally, a sigmoid activation function is applied to produce the segmentation mask, where each pixel value represents the probability of that pixel belonging to the ROI. Figure 5 shows the architecture of the proposed mask decoder.

3.3. Training Procedure

We trained SALM using a combination of binary cross-entropy (BCE) loss and Dice loss. The BCE loss measures the pixel-wise difference between the predicted segmentation mask and the ground truth mask, while the Dice loss measures the overlap between the two. The total loss is a weighted sum of the two losses, as follows:

$$\mathrm{Loss}=\alpha ·\mathrm{BCE}\mathrm{Loss}+\beta ·\mathrm{Dice}\mathrm{Loss}$$

(5)

where $\alpha$ and $\beta$ are hyperparameters that control the relative importance of the two losses. In our experiments, we set $\alpha$ = 0.5 and $\beta$ = 0.5, giving equal weight to both losses.

We used the Adam optimizer [45] with a learning rate of 1 × ${10}^{-4}$ and a batch size of 8 for 2D segmentation and 1 for 3D segmentation due to the larger memory requirements of processing 3D volumes. The model was trained for 100 epochs, and the best-performing model on the validation set was selected for evaluation on the test set.

To augment the training data, we applied random horizontal and vertical flips, random rotations (−20 to +20 degrees), and random scaling (0.8 to 1.2) to the input images and corresponding masks during training.

The model was implemented using the PyTorch 2.0 framework [46] and trained on a single NVIDIA A100 GPU [47].

4. Results and Discussion

4.1. Quantitative Results

We evaluated SALM’s performance on both 2D and 3D segmentation tasks using the LUNA16 dataset. For 2D segmentation, we report the accuracy, the Area under the ROC Curve (AUC), and the Dice Similarity Coefficient (DSC) on the test set. For 3D segmentation, we report the F1-score and DSC, following the evaluation protocol commonly used in previous studies [12,48,49,50,51].

Table 2. Comparison of SALM with state-of-the-art methods on 2D segmentation using the LUNA16 dataset.

Model	Accuracy	AUC	DSC
SegNet [52]	95.00	N/A	N/A
UNET [52]	91.00	N/A	N/A
FCN [53]	N/A	N/A	92.00
DeepLabV3 [54]	94.90	N/A	N/A
SALM	99.00	99.00	93.00

N/A = not available; In bold: best result.

shows the results of SALM on the 2D segmentation task compared with several state-of-the-art methods. SALM achieves an accuracy of 99%, an AUC of 99%, and a DSC of 93%, demonstrating its strong performance in accurately segmenting ROI in 2D slices.

Table 3. SALM comparison with different architectures for Lung CT Segmentation in 3D.

Model	F1-Score	DSC	Test Size	Database (No. of Patients)
FCN V.Net [48]	N/A	80.00	32	96
Hybrid CNN [49]	N/A	58.00	N/A	260
Recurrent 3D-DenseUNet [50]	N/A	72.28	N/A	300
FoL [51]	N/A	83.68	430 slices	40
SAM [12]	42.83	43.95	174	-
SALM-3D	75.57	81.88	174	888

N/A = not available; In bold: best result.

shows the results of SALM on the 3D segmentation task compared with several state-of-the-art methods. SALM achieves an F1-score of 75.57% and a DSC of 81.88% on the test set of 174 3D images (124,662 slices). These results demonstrate SALM’s ability to effectively leverage 3D spatial information for accurate ROI segmentation in lung CT volumes. Notably, SALM outperforms the original SAM [12] evaluated on our 3D test set, highlighting the importance of our modified positional encoding for capturing 3D spatial relationships.

To further evaluate the effectiveness of our modified positional encoding, we conducted an ablation study comparing the performances of SALM with and without the $3\pi$ modification.

Table 4. Ablation study on the effect of the modified positional encoding.

Positional Encoding	F1-Score (%)	DSC (%)
Standard	56.23	59.53
Modified (3$\pi$)	75.57	81.88

In bold: best result.

shows the results of this study on the 3D segmentation task. We observe that the modified positional encoding leads to a significant improvement in both F1-score and DSC, demonstrating its importance for capturing multi-scale spatial relationships in 3D volumes.

4.2. Qualitative Results

In addition to the quantitative results, we also present qualitative results to visually demonstrate the effectiveness of SALM in segmenting ROI in lung CT images. Figure 6 and Figure 7 show example segmentation results of SALM applied to 2D slices from the LUNA16 test set. These results demonstrate that SALM is able to accurately segment ROI of varying sizes and shapes.

However, we also observed some cases where SALM produced less accurate segmentations, particularly for very small ROI or ROI with irregular shapes (Figure 8).

4.3. External Validation on PleThora Dataset

To further assess the generalizability of SALM, we evaluated the model on 245 DICOM CT scans of the publicly available PleThora dataset [15,16]. PleThora (Pleural Effusion and Thoracic Organ Segmentation) is a completely external dataset comprising diseased lung CT scans, specifically designed for benchmarking chest CT processing pipelines and focusing on the challenging task of segmenting pleural effusions and thoracic organs in pathological cases. This dataset provides a valuable independent validation of SALM’s performance on data beyond the LUNA16 dataset used for training and initial evaluation.

The inference was performed on a single NVIDIA GeForce GTX 1080 Ti GPU [55] with 12 GB of memory, with an average processing time of approximately 1.68 images per second. To further assess the model’s efficiency across different hardware, we also conducted inference on an Apple MacBook Pro (2024) equipped with an M3 chip and 16 GB of memory [56], achieving an average processing time of approximately 1.92 images per second. A detailed comparison of processing times across these platforms is presented in

Table 5. Inference time comparison across different GPUs on 245 CT scans.

GPU	Processing Time (Minutes)	Images per Second
NVIDIA GeForce GTX 1080 Ti (12 GB)	2.25	1.68
Apple M3 (16 GB)	7.51	1.92

. The quantitative results on the PleThora dataset are summarized in

Table 6. SALM performance on PleThora dataset (external validation).

Metric	Value
Mean Dice Score	0.7882 ± 0.1514
Mean Accuracy	0.9764 ± 0.0146
AUC-ROC	0.8592

.

The results on the PleThora dataset demonstrate robust segmentation performance, achieving a mean Dice score of 78.82% and a high accuracy of 97.64%. An AUC-ROC of 85.92% further indicates good discriminatory power of the model’s segmentation. While the Dice score is slightly lower compared with the results on the LUNA16 dataset (81.88% in 3D segmentation), it is important to consider that PleThora presents a more challenging scenario due to the presence of diseased lungs and pleural effusions, which introduce greater anatomical variability and complexity (Figure 9). Furthermore, a Dice score of 78.82% on a completely external and challenging dataset represents a strong and clinically relevant performance, demonstrating SALM’s ability to generalize beyond the training data and maintain good accuracy in more complex, pathological lung CT scans. Additionally, Figure 10 highlights how SALM, while processing 3D volumes slice by slice, effectively segments lung ROI in a 2D viewing format, which is typical in clinical practice for lung CT analysis. The 2D slice visualization allows clinicians to readily interpret the segmentation results within their familiar diagnostic workflow, demonstrating the practical applicability of SALM for 3D lung CT data when viewed and analyzed in standard 2D slice-based displays.

To further illustrate the impact of our modified positional encoding, Figure 11 presents a qualitative comparison of SALM’s segmentation performance on a PleThora dataset slice, with and without the $3\pi$ modification. The top row of Figure 11 shows the segmentation using the standard positional encoding, while the bottom row shows the result with our modified positional encoding (SALM). As we can observe in the “Prediction Errors” panels, the model using the standard positional encoding exhibits a significantly larger overpredicted area (shown in red), indicating a tendency to falsely identify regions outside the ground truth ROI. In contrast, SALM, with the modified positional encoding, substantially reduces this overprediction, achieving a more precise segmentation that aligns more closely with the ground truth mask. While SALM still shows some limitations in segmenting the diseased portion of the lung in this challenging PleThora example, as indicated by the missed areas (blue), the visual comparison clearly demonstrates the effectiveness of our modified positional encoding in improving segmentation precision and reducing noise-induced false-positive detections, particularly in complex pathological lung tissue. This qualitative finding complements the quantitative results of our ablation study (Table 4).

4.4. Discussion

The quantitative and qualitative results demonstrate that SALM is a promising model for both 2D and 3D ROI segmentation in lung CT images. The model’s good performance can be attributed to several factors, including the use of the Vision Transformer architecture, FM weights, the proposed adaptation of positional encoding, and the effective training procedure.

The ViT architecture enables SALM to capture long-range dependencies within the input images, which is crucial for understanding the global context of lung CT scans. The modified positional encoding further enhances the model’s ability to capture multi-scale spatial relationships, particularly in 3D volumes, by providing explicit information about the location of each slice within the volume.

The ablation study demonstrates the significant impact of our modified positional encoding on the model’s performance. By increasing the spatial frequency of the encoding, we enable the model to capture finer-grained positional information, leading to improved segmentation accuracy.

The training procedure, which combines BCE loss and Dice loss, effectively balances the need for pixel-wise accuracy and overall segmentation quality. Data augmentation techniques further improve the model’s robustness and generalization ability.

As shown in Figure 9, SALM demonstrates a degree of robustness in segmenting diseased lungs, successfully capturing the overall lung regions despite the presence of significant anatomical variability and heterogeneous tissue densities. However, the segmentations on PleThora also exhibit increased roughness and less smooth boundaries, particularly in areas affected by disease. This suggests that while SALM generalizes reasonably well to pathological cases, the segmentation accuracy is impacted by the increased complexity and heterogeneity of diseased lung tissue compared with the relatively cleaner LUNA16 dataset. The model tends to include diseased areas within the segmented lung regions, which, depending on the clinical application, might be considered either appropriate or requiring further refinement to specifically delineate only healthy lung parenchyma. These qualitative observations on PleThora are consistent with the slightly lower Dice scores obtained on this dataset compared with LUNA16, further emphasizing the challenges of lung segmentation in pathological contexts and highlighting areas for future improvement in model robustness and refinement.

Despite its good performance, SALM has some limitations. The model’s accuracy can be affected by the quality of the input images, particularly in cases of low contrast or high noise. Additionally, the model may struggle with very small ROI or ROI with highly irregular shapes, as observed in some of the qualitative results. The overpredicted areas (false positives) can be attributed to the following specific technical and anatomical factors:

• Tissue Density Similarity: Anatomical structures with attenuation coefficients (Hounsfield units) similar to the targeted ROI can induce classification errors. This densitometric ambiguity is particularly pronounced in transition zones between soft tissues and vascular structures. For example, blood vessels, scar tissue, and areas of fibrosis can exhibit Hounsfield values that overlap with those of lung nodules, especially part-solid or ground-glass nodules. This overlap makes it challenging for the model to solely rely on intensity information for accurate differentiation, leading to false-positive detections in regions with dense vasculature or fibrotic changes.
• Morphological Characteristics: The presence of anatomical structures with geometric characteristics similar to irregular or hidden ROI can generate segmentation errors. The model may incorrectly interpret these formations as potential ROI. In advanced lung disease, ROI such as tumors or affected lung parenchyma can exhibit highly irregular shapes and may be partially obscured or hidden by surrounding diseased tissue, pleural effusions, or consolidation, as frequently observed in the PleThora dataset. SALM, trained primarily on datasets with relatively well-defined ROI like LUNA16, may struggle to accurately delineate these highly irregular and poorly defined ROI in advanced pathological cases. The model’s reliance on shape features learned from less complex examples might lead to under-segmentation or inaccurate boundary delineation when confronted with the complex and atypical morphologies characteristic of advanced lung pathologies. This is further compounded by the fact that, in diseased lungs, the contrast between the ROI and the surrounding abnormal tissue may be reduced, making morphological differentiation even more challenging for the model.
• Positional Encoding Limitations: Overpredictions can result from an imperfect capture of three-dimensional spatial relationships, despite the use of modified positional encoding. Peripheral areas of anatomical structures, where intensity gradients are less pronounced, are particularly susceptible to misclassification. While our modified 3D positional encoding enhances spatial awareness, it is not perfect. In regions where anatomical structures are complex or boundaries are ill-defined, particularly at the periphery of organs or lesions where intensity transitions are gradual, the model might struggle to precisely delineate the ROI. This can lead to over-segmentation extending into surrounding tissues or misclassification of adjacent structures due to limitations in fully capturing the intricate 3D spatial context.
• Acquisition Artifacts: Artifacts inherent to CT imaging, such as beam hardening or partial volume effects, can create intensity variations that mislead the model, particularly at interfaces between different tissue types. Beam hardening artifacts, often seen near dense bone structures like ribs, and partial volume averaging, especially in regions with thin slices or complex anatomy, can introduce artificial intensity gradients and noise. These artifacts can create spurious features that the model might misinterpret as relevant structures or boundaries, leading to false-positive detections or inaccurate segmentation, especially at the lung periphery or near the chest wall.

This observation about false positives emphasizes the importance of optimizing the model’s attention mechanisms and incorporating stricter anatomical constraints in the segmentation architecture.

We also acknowledge that further refinements are possible through post-processing techniques. Specifically, methods such as hole filling [57,58] and morphological operations [59] hold the potential to improve the segmentation masks further by addressing common imperfections in segmentation outputs. Hole filling can effectively close small gaps and cavities within the segmented regions, leading to more complete and solid object masks, which is particularly useful for lung nodules that may appear with internal low-density areas. Morphological operations, such as erosion and dilation, can smooth out jagged boundaries, remove small spurious pixels or noise outside the main object, and refine the overall shape of the segmented ROI to better align with anatomical structures.

Future work could focus on addressing these limitations and further enhancing SALM’s capabilities and clinical utility. Specifically, the following promising avenues for future research could be explored:

• Refinement for Small, Irregular, and Hidden ROI: To improve segmentation accuracy for ROI that are not only small and irregularly shaped but also poorly defined or hidden due to the complexities of advanced lung disease, future work could focus on incorporating more advanced data augmentation techniques, such as localized deformations and synthetic ROI generation [60], specifically targeting the variability in size, shape, and visibility of lesions, particularly in advanced pathological cases like those observed in the PleThora dataset. Furthermore, research could investigate the incorporation of attention mechanisms that are more sensitive to fine-grained details and boundary information, potentially through novel Transformer architectures or modifications to the existing self-attention layers, to better capture the indistinct and ambiguous boundaries of hidden or irregular ROI. Finally, exploring loss functions that explicitly emphasize boundary sharpness and penalize over-segmentation, such as boundary loss [61] or contour loss [62], could further improve delineation accuracy for challenging ROI with complex and ill-defined borders, often encountered in diseased lung tissue.
• Addressing Over-segmentation and False Positives: To challenge over-segmentation and reduce false-positive detections, particularly arising from tissue density similarity and acquisition artifacts, future research could explore the incorporation of anatomical priors and contextual information into the model architecture. This might involve integrating anatomical segmentation maps [63] as additional input channels or exploring hierarchical segmentation approaches [64] that first segment broader anatomical regions before refining the ROI segmentation. Furthermore, exploring contrastive learning techniques [65] to better discriminate between ROI and similar-appearing background structures could enhance the model’s robustness to densitometric ambiguities.
• Exploration of Advanced Architectures for Efficiency and Performance: To further improve both the efficiency and the performance of SALM, future work could investigate the integration of novel Transformer architectures. Specifically, the exploration of xLSTM [66], with its ability to process sequential data efficiently across multiple timescales, could be beneficial for modeling temporal dependencies in longitudinal studies or dynamic contrast-enhanced CT. Mamba [67], with its selective state space model, offers a promising avenue for capturing long-range dependencies in both spatial and temporal dimensions with high computational efficiency, potentially allowing for more efficient processing of large 3D volumes. The integration of these advanced architectures, combined with the modified positional encoding scheme described in this paper, could lead to significant improvements in segmentation accuracy, robustness, and clinical utility, paving the way for real-time applications and deployment on resource-constrained devices.
• Incorporation of Clinical and Demographic Data: To further enhance the clinical relevance and diagnostic potential of SALM, future work could explore the possibility of incorporating additional clinical information, such as patient demographics, medical history, and radiology reports, as auxiliary inputs to the model. This could enable SALM to learn more nuanced relationships between image features and clinical context, potentially improving segmentation accuracy and enabling more informed diagnostic predictions or risk stratification.

5. Conclusions

In this paper, we presented SALM, a novel deep learning model for 2D and 3D ROI segmentation in lung CT images. SALM leverages the Vision Transformer architecture and incorporates a unique adaptation of positional encoding that enables it to effectively capture multi-scale spatial relationships within 3D volumes. We evaluated SALM on the publicly available LUNA16 dataset and demonstrated its strong performance in both 2D and 3D segmentation tasks. We further validated SALM’s robustness and generalizability by testing it on the completely external and clinically challenging PleThora dataset of diseased lung CT scans.

The quantitative results show that SALM achieves state-of-the-art performance on LUNA16, with an AUC of 99% and a DSC of 93% for 2D segmentation and an F1-score of 75.57% and a DSC of 81.88% for 3D segmentation. The ablation study further highlights the importance of our modified positional encoding, which leads to significant improvements in segmentation accuracy. Moreover, the external validation on PleThora, achieving a mean Dice score of 78.82% on diseased lungs, provides strong evidence for SALM’s ability to generalize to unseen data and maintain clinically relevant performance in more complex pathological scenarios. The qualitative results demonstrate that SALM is able to accurately segment ROI of varying sizes, shapes, and locations, in both 2D slices and 3D volumes, with visual examples on PleThora highlighting both the model’s robustness and the challenges posed by diseased lung tissue. However, some challenges remain, particularly in segmenting very small nodules or nodules with irregular shapes.

Future work could focus on addressing the remaining limitations and further expanding SALM’s capabilities to enhance its clinical utility and robustness. Specifically, future research could prioritize refinement for small, irregular, and hidden ROI, exploring advanced data augmentation techniques, more sensitive attention mechanisms, and specialized loss functions to improve delineation accuracy in challenging pathological contexts. Furthermore, to challenge over-segmentation and false positives, incorporating anatomical priors, contextual information, and contrastive learning strategies could be investigated. Expanding SALM’s generalizability beyond lung CT is another interesting direction, with future work focusing on validation across diverse datasets and modalities like PET-CT and MRI, potentially through fine-tuning and architectural adaptations for multimodal input. To improve both efficiency and performance, the exploration of novel architectures such as xLSTM and Mamba holds significant promise, potentially enabling real-time applications and deployment on resource-constrained devices. Finally, incorporating clinical and demographic data as auxiliary inputs could further enhance SALM’s clinical relevance and diagnostic potential, leading to more nuanced and context-aware segmentation and potentially enabling downstream diagnostic or prognostic tasks.