Evaluating Deep Learning Architectures for Breast Tumor Classification and Ultrasound Image Detection Using Transfer Learning

Abstract

The intersection of medical image classification and deep learning has garnered increasing research interest, particularly in the context of breast tumor detection using ultrasound images. Prior studies have predominantly focused on image classification, segmentation, and feature extraction, often assuming that the input images, whether sourced from healthcare professionals or individuals, are valid and relevant for analysis. To address this, we propose an initial binary classification filter to distinguish between relevant and irrelevant images, ensuring only meaningful data proceeds to subsequent analysis. However, the primary focus of this study lies in investigating the performance of a hierarchical two-tier classification architecture compared to a traditional flat three-class classification model, by employing a well-established breast ultrasound images dataset. Specifically, we explore whether sequentially breaking down the problem into binary classifications, first identifying normal versus tumorous tissue and then distinguishing benign from malignant tumors, yields better accuracy and robustness than directly classifying all three categories in a single step. Using a range of evaluation metrics, the hierarchical architecture demonstrates notable advantages in certain critical aspects of model performance. The findings of this study provide valuable guidance for selecting the optimal architecture for the final model, facilitating its seamless integration into a web application for deployment. These insights are further anticipated to advance future algorithm development and broaden the potential of the research applicability across diverse fields.

1. Introduction

The integration of artificial intelligence into various aspects of modern life, coupled with unprecedented advancements in computational resources such as processors and graphics cards, has driven remarkable progress across numerous domains, including biomedical technology. The recognition and classification of diseases using machine learning algorithms through the use of either simple numerical or complex data, such as images, have been successfully applied for many years, particularly in tumor identification [1]. Specifically, numerous studies have focused on breast cancer detection, which remains one of the leading causes of death among women worldwide [2]. Building on this foundation, breast cancer diagnosis through medical imaging and machine learning has emerged as a critical component of modern healthcare, fueled by growing interest in the application of artificial intelligence to this field. Studies have consistently shown that early detection significantly improves survival rates among women [3], and models, whether designed to assist doctors or operate autonomously, are aimed at improving early diagnoses while alleviating the workload of medical professionals.

The motivation behind this study arises from practical challenges encountered during the development of a real-world web application. This research explores the potential of hierarchical and flat classification approaches to identify the most effective method for breast tissue classification, specifically examining whether a two-branch binary architecture can outperform the traditional three-class classification paradigm. Additionally, the integration of a filtering layer designed to distinguish between relevant and irrelevant input images aims to enhance system reliability and input relevance. The goal of this study is to determine whether the development of advanced binary classification systems represents a more effective strategy than optimizing traditional three-class classification algorithms, thereby contributing to the advancement of diagnostic methodologies in healthcare and the field of image recognition.

2. Related Work

Breast cancer detection has seen significant advancements through the application of machine learning and imaging techniques in recent years. Most existing studies have primarily focused on using mammography images to extract useful features, whether for tumor detection, segmentation, or directly predicting the malignancy of the identified tumor based on the input image [4,5]. However, one challenge is that some hospitals and medical centers may lack the necessary equipment for digital mammography. Additionally, in cases where women have dense breast tissue, mammography may fail to detect cancers [6]. A common supplementary or alternative imaging modality in such scenarios is breast ultrasound, which, despite being prone to noise, is a safe, cost-effective, and portable method frequently employed by radiologists [1].

Convolutional Neural Networks (CNNs) have long been the dominant deep learning approach for breast ultrasound analysis, with numerous architectures demonstrating strong performance on their respective datasets [7,8,9]. While Transformer-based vision models with attention mechanisms have recently emerged as promising alternatives [10], CNNs remain the industry standard, with research primarily emphasizing deeper architectures and greater complexity to extract more intricate features from ultrasound images [11]. This approach seeks to improve the accuracy of predictions for the three-class classification task of distinguishing between malignant, benign, and normal cases. While traditional flat classification approaches, such as those referenced, dominate medical imaging tasks, hierarchical classification architectures have garnered attention for their potential to enhance both accuracy and interpretability by breaking down complex problems into simpler subtasks [12,13]. However, the application of such architectures to breast ultrasound images remains underexplored, presenting an opportunity to investigate their benefits over flat three-class models.

Despite nonstop advancements over the years, the field faces persistent challenges, particularly concerning the interpretability of CNN models and the computational demands associated with training complex architectures [14]. Large CNN architectures, which are typically designed to classify multiple categories, often demand significant computational resources and can take hours or even days to train on extensive datasets using Graphics Processing Units (GPUs). Building on these constraints, along with the limited availability of medical data largely restricted by patient privacy concerns, many recent studies have adopted transfer learning as an effective approach [6]. In transfer learning, a pretrained model, along with its parameters, weights, and gained knowledge from a large dataset, is repurposed to perform a different task than originally intended [15]. Hyperparameter tuning is also a critical aspect of this process, as it optimizes the performance of the model for the specific task and dataset, ultimately enhancing its accuracy [16].

On an application level, numerous studies have focused primarily on mammography or histopathology images, as these datasets are open-source and more readily available. One such application is ABCanDroid [17], which utilizes a high-accuracy ResNet-101 model with transfer learning in an Android environment. Another notable example is Branet [18], a newly developed mobile application that integrates both segmentation and classification techniques for mammography and ultrasound images to predict whether a tumor is benign or malignant. While these applications are distinguished by their strong performance, their scalability when incorporating new data remains a concern, particularly in cases where unrelated images are input into the models.

3. Methodology

The objective of this study is to assess the performance of a hierarchical two-tier binary classification architecture in comparison to the traditional three-class classification approach, which has been widely adopted in previous studies. Despite limited application in medical data, hierarchical architectures have demonstrated superior performance compared to flat models in image classification across various domains [13]. Notably, they have been successful in analyzing small bowel biopsy images [13] and distinguishing between healthy eyes and those affected by glaucoma, including classifying four glaucoma subtypes [19]. To further address the challenges identified on an application level, after determining the superior method, a set of models will be trained and evaluated to serve as the filter stage. These models will be designed to exclude irrelevant images, ensuring that only diagnostically relevant data progresses to the subsequent stages of analysis. The two approaches, which will be compared and evaluated following the filter level, are presented in Figure 1.

In the literature on hierarchical classification, the first approach depicted in the image is known as the Local Classifier per Node (LCN) approach, which is by far the most widely used method in hierarchical architectures [20,21]. This technique involves training a binary classifier for each node within the class hierarchy, progressing from the root to the leaf nodes. The second approach, also illustrated in the image, is the flat classification method, commonly referred to in earlier studies as the Global Classifier [22]. Unlike LCN, this approach disregards the class hierarchy entirely, focusing solely on predicting the classes at the leaf nodes. While this method simplifies the classification task, it does so at the expense of leveraging hierarchical relationships. Both approaches offer distinct advantages and disadvantages when applied to classification tasks. However, their comparative effectiveness in the context of breast ultrasound image classification remains to be evaluated.

4. Data Sources and Image Preprocessing

The experimental analysis was conducted using two primary datasets. The first and most critical dataset, utilized for classifying benign, malignant, and normal breast tissues in the final layers of the three hierarchical architectures, is the widely recognized benchmark Breast Ultrasound Images Dataset [23]. This dataset comprises 780 ultrasound images, including 133 classified as normal, 437 as benign, and 210 as malignant, along with their corresponding tumor segmentation mask images. However, it has been consistently noted across multiple sources that in addition to the class imbalance present in the Breast Ultrasound Images Dataset, the number of images available for training is insufficient for effective model performance [9,24]. Studies have also shown that applying image augmentation techniques not only addresses this limitation, but also leads to improved classification performance [25,26,27]. Consequently, an image augmentation step was deemed essential to enhance the dataset and improve the training process. Representative samples from this dataset, illustrating its diverse image categories, are presented in Figure 2.

For the filter layer, designed to eliminate irrelevant images, a combination of publicly available datasets was utilized. To represent random irrelevant images, the “Unsplash Random Images Collection” from Kaggle, featuring 802 images of landscapes, animals, and street scenes, was used alongside four medical datasets: the “Brain MRI Images” dataset with 14,700 images, the “Mammography Small Dataset” with 106 mammograms, the “Chest X-Ray Images” dataset [28] containing 5863 normal and pneumonia chest X-rays, and the “BreaKHis” dataset [29] with 780 biopsy images of benign and malignant breast tumors.

In the image preprocessing stage for the first dataset, segmentation mask images were discarded, as they were not relevant for classification. All malignant, benign, and normal images were horizontally mirrored, doubling the dataset to 1560 images. However, an imbalance of 56%, 26%, and 18% favoring benign images remained, which could affect model performance particularly in tasks involving pattern recognition and classification [30]. To address this, data augmentation techniques were applied to reach 1150 images per class, generating 276 benign, 730 malignant, and 884 normal images. A portion of the existing images was randomly selected for cropping/zooming with a 1.2× factor. Common distortions in medical imaging, such as blurring, contrast variations, and noise, which can affect diagnostic accuracy [31], were simulated across the augmented dataset. Some images had their contrast reduced, others were enhanced using Contrast Limited Adaptive Histogram Equalization (CLAHE), and others still were subjected to Gaussian blur or salt-and-pepper noise. The zooming/cropping step was applied to normal images after other augmentations due to their smaller quantity. Figure 3 shows the applied distortion types during the ultrasound image data augmentation process.

In architectures such as the ones examined in this study, algorithms trained on imbalanced datasets commonly demonstrate a tendency to favor the majority classes [21]. To mitigate this issue and achieve balance with the 3460 breast ultrasound images, the irrelevant image dataset was appropriately adjusted. After eliminating duplicates, the dataset was augmented primarily using the “Unsplash Random Images Collection”, employing techniques such as mirroring, random cropping, and converting images to black and white, which increased the dataset size to 2184 images. Additionally, 350 images were randomly selected and prepared from the remaining four medical datasets. In the case of the mammography dataset, all 212 images, after mirroring, were included, resulting in a final dataset of 3446 samples.

5. Selected Models for Evaluation

The three pretrained models selected for evaluation within the two proposed architectures were chosen based on their established prominence and frequent application in transfer learning for ultrasound imaging in recent years [27]. These models are recognized for their ability to deliver high levels of accuracy and efficiency during training, making them well-suited for tasks involving complex feature extraction and precise classification.

5.1. VGG-16

When discussing deep learning architectures, one of the most prominent and foundational Convolutional Neural Networks is VGG-16. This architecture gained recognition in the 2014 ImageNet Challenge, securing first place in the localization task and achieving second place in classification. Structurally, VGG-16 comprises 13 convolutional layers and 3 fully connected layers, distinguished from earlier models by its consistent use of small 3 × 3 filters in all convolutional layers, which enhances feature extraction efficiency and overall performance [32]. VGG-16 has demonstrated strong performance in transfer learning applications for breast cancer classification, both with and without data augmentation [33,34]. Furthermore, it has shown high accuracy in distinguishing relevant from irrelevant images, making it effective for filtering social media content related or unrelated to natural diseases [35].

5.2. InceptionV3

Together with ResNet [36], the Inception architecture [37] introduced a transformative approach in computer vision, challenging the trend of deepening VGG-based architectures by effectively addressing the vanishing gradients problem [38]. This issue, where network derivatives diminish and impede learning from new data, was mitigated by the structure of Inception models. Known for its unique multi-scale feature extraction capabilities, the Inception architecture achieved success by employing layers with varying filter sizes within each module. In later years, this design further reduced the parameter count and incorporated regularization techniques, including batch-normalized auxiliary classifiers and label smoothing, allowing efficient training even with moderate-sized datasets [39]. InceptionV3, in particular, has demonstrated robust performance in transfer learning applications on the Breast Ultrasound Images Dataset (BUSI) utilized in this study [40].

5.3. NASNet

Unlike earlier models with architectures grounded in traditional design principles for deepening neural networks, NASNet, developed by the Google Brain team, introduces a unique approach derived from Neural Architecture Search (NAS). In this process, a Recurrent Neural Network (RNN) controller explores the architecture space, initially training and optimizing candidate models on the CIFAR-10 dataset. The best-performing architecture is then scaled and adapted for the larger ImageNet dataset, ensuring efficient feature extraction and optimal performance [41]. The structure of NASNet is composed of convolutional layers interspersed with pooling layers, enhancing feature extraction across varying levels of representation. When applied to the Breast Ultrasound Images Dataset with data augmentation techniques, NASNet has demonstrated superiority in Transfer Learning (TL) applications, outperforming many other models in this domain [23,42].

6. Training Process

This section provides a comprehensive overview of the training process, including a detailed description of the hardware and software employed in conducting the experiments, the normalization techniques applied to the data, and the final adjustments made to the hyperparameters of the networks to optimize performance prior to training.

6.1. Experimental Setup

The proposed models, architectures and final application were implemented on an NVIDIA DGX Workstation (supplied by NVIDIA Corporation, Santa Clara, CA, USA) equipped with a 2.2 GHz Intel Xeon E5-2698 v4 CPU featuring 20 cores (Intel Corporation, Santa Clara, CA, USA), four Tesla V100 Tensor Core GPUs (NVIDIA Corporation, Santa Clara, CA, USA), and 256 GB of RDIMM DDR4 RAM. The software environment included Python 3.10.12, with TensorFlow 2.15.0 and Keras 2.15.0 for model development, alongside supporting libraries such as scikit-learn 1.4.2, Matplotlib 3.10.1, OpenCV 4.11.0, Pillow 11.1.0, and Seaborn 0.13.2.

6.2. Data Normalization

The data normalization step is crucial for improving processing speed, accuracy, and feature extraction in neural networks. Although data augmentation and cleaning enhance the dataset, it may still fall short in providing sufficient, relevant information, potentially complicating model training across epochs instead of improving it [43]. After augmentation, data for each classification level in the hierarchical architectures are divided into 70% training, 15% validation, and 15% test sets. Pixel values are then normalized to a 0 to 1 range by dividing each pixel by their maximum value (255), and finally, images are resized to meet the dimensional requirements of each model.

6.3. Fine Tuning

After data augmentation and normalization, each model is initialized as a base model using pretrained models of TensorFlow, reducing susceptibility to errors and eliminating the need to train models from scratch, saving both time and computational resources [44]. The VGG-16, InceptionV3, and NASNet models are loaded with weights from the ImageNet dataset, excluding their final layers. For VGG-16, the final layers consist of a Flatten layer and a Dense layer with 256 neurons using a ReLU activation function. In InceptionV3 and NASNet, the final layers include a 2D Global Average Pooling layer followed by a Dense layer with 1024 neurons, also using ReLU activation. With the inclusion of the extra layers, the total number of parameters for VGG-16 is 21,137,729, for InceptionV3 the total number is 23,901,985, and for NASNet the total number is 89,047,635, demonstrating the expanded complexity of each model. The final activation layer for each model is adjusted based on the classification task: for three-class classification, a softmax layer with three units is used, while binary classification uses a single sigmoid unit. In all models, early layers are frozen to retain pretrained weights for foundational feature extraction. For the high-level task of irrelevant/relevant image classification, a grid search was performed using the AdamW, Adam, Adamax, and SGD optimizers with learning rates of 10⁻², 10⁻³, 10⁻⁴ and 10⁻⁵. The results indicated that the Adam optimizer with a learning rate of 10⁻³ was the most effective choice for every scenario. This configuration, along with 5 training epochs, proved ideal for achieving fast convergence. For the breast tumor classification task, hyperparameters were selected based on the findings of Al-Dhabyani et al. [24], who successfully trained models for 10 epochs using the Adam optimizer with a learning rate of 10⁻³, yielding satisfactory results with or without data augmentation. Batch sizes for all image generators were consistently set to 32.

The three models within each level of the two architectures are trained using the training data, with their performance evaluated based on the corresponding validation data. By leveraging TensorFlow’s history object and the Matplotlib library, visualizing the changes in accuracy and loss for both the training and validation sets across each epoch becomes straightforward. This visualization is essential for monitoring the training process and identifying potential issues, such as overfitting or underfitting, that may impact each model [45]. Figure 4, Figure 5 and Figure 6 depict the variations in accuracy and loss over epochs for both training and validation sets, offering valuable insights into how the three models perform at each level of the two architectures.

7. Evaluation

After training the three models, their performance was evaluated on the previously reserved testing set to assess each model’s ability to correctly classify each class. It is important to note that the training, validation, and testing datasets for all architectures tasked with either binary or three-class classification of breast ultrasound images were consistently fixed across all experiments. This approach ensures that the same data subsets were used for both training and evaluation, thus eliminating potential inaccuracies that could arise from randomly selecting data in each iteration. Based on the True Positive (TP), False Positive (FP), True Negative (TN), and False Negative (FN) values obtained from each model’s confusion matrix, common classification metrics such as accuracy, precision, recall (sensitivity), and F₁ score were used to assess model performance at each stage of the architectures. Precision and recall were specifically calculated for the class of interest at each level: relevant images for the filter layer, and malignant/tumorous images for the later layers. The formulas for these metrics are outlined as follows:

$$Accuracy={\displaystyle \frac{TP+TN}{T}}$$

(1)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(2)

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(3)

$$F_{1}score=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$$

(4)

To assess whether the observed differences in model performance across different training processes and data splits are meaningful and not due to random variation, a paired t-test was conducted. The paired t-test, also known as Student’s t-test, is a well-established statistical method that evaluates whether the mean difference between two related groups is statistically significant [46]. By comparing the observed t-statistic to the t-distribution, the p-value which quantifies the likelihood that the observed differences occurred by chance, is calculated. A p-value smaller than a predefined threshold (typically 0.05) suggests that the differences are statistically unlikely to have arisen randomly, supporting the hypothesis that the observed performance disparity is meaningful [47]. The equation for calculating the paired t-test is provided below:

$$t={\displaystyle \frac{\frac{{\sum }_{i=1}^n{d}_i}{n}}{s_d/\sqrt{n}}}$$

(5)

where d_i represents the difference of the pair values for the i-th observation, n is the number of paired observations, and s_d is the standard deviation of the paired differences. In this study, the paired t-test was specifically applied to compare the accuracy metric between the two best performing models, in order to confirm the significance of their performance.

8. Results

Before presenting the final results and comparing the two architectures, it is important to note that previous studies have demonstrated the superior performance of NASNet in binary classification tasks involving breast tumor data through transfer learning [48], particularly in distinguishing between benign and malignant cases. Based on these findings, NASNet was chosen as the model for the final layer of the two-level binary classification architecture. For the comparison of the three models inside this architecture, the final layer remains fixed as NASNet, while the layer responsible for binary classification between tumorous and non tumorous tissue varies across the architectures.

The testing results of the two architectures, along with the integrated models following the training process, are presented in

Table 1. Performance metrics of the two architectures.

Architecture	Model	Accuracy	Sensitivity	Precision	F1 Score	Training Time	Inference Time
Two level	VGG-16 & NASNet	89.1	89.1	91.1	90.1	11 min	120 ms
binary	InceptionV3 & NASNet	86.8	83.2	94.7	88.6	11 min	130 ms
classification	NASNet & NASNet	92.7	91.9	92.4	92.1	15 min	200 ms
Flat three	VGG-16	88.7	97.1	80.7	88.1	5 min	30 ms
class	InceptionV3	87.3	83.8	93.5	88.4	5 min	40 ms
classification	NASNet	93.1	88.4	95.1	91.6	7 min	80 ms

. This table provides a detailed comparison, including the evaluation metrics discussed earlier along with the training time required for each model, and the time taken to generate a prediction for a single image, offering a comprehensive assessment of their performance and efficiency.

Following the detailed comparison presented in Table 1, the best-performing model combination from Architecture 1 and the most accurate flat three-class classification model were selected to generate confusion matrices for each, as presented in Figure 7. The confusion matrix, often referred to as the error matrix, is a well-established tool used to visualize the performance of a classification algorithm. It offers valuable insights into the types of errors made by the model, facilitating the calculation of various evaluation metrics [49].

The performance metrics for the models in the filter layer, which distinguishes between irrelevant and relevant images, are shown in

Table 2. Performance metrics for the models in the filter layer.

Model	Accuracy	Sensitivity	Precision	F1 Score	Training Time	Inference Time
VGG-16	99.8	99.9	99.9	99.9	4.3 mins	30 ms
InceptionV3	99.5	99.4	99.6	99.5	4.0 mins	40 ms
NASNet	99.9	100	99.8	99.9	6.0 mins	80 ms

. This table includes evaluation metrics such as training and inference times for each model, similar to the data presented in Table 1.

9. Discussion

The results presented in Table 1 highlight that the "NASNet & NASNet" architecture is the most effective choice for the binary classification strategy of two levels, achieving an accuracy of 92.7%. Although not explicitly shown in Table 1, this decision was primarily guided by the performance of the three algorithms when trained to distinguish between normal breast tissue and a combined category of malignant and benign tissue. In this binary task, NASNet exhibited exceptional performance, achieving an impressive accuracy of 97.6% and a sensitivity of 97.8%. Regarding the three-class classification architecture, the NASNet model achieved the highest accuracy of 93.1%, demonstrating its reliability in handling multi-class classification scenarios. Ten training sessions were conducted on distinct random data splits to ensure that the accuracy of the two models was not the result of chance. The models showed consistent performance across runs, and a paired t-test yielded a p-value of 0.04, which is below the conventional significance threshold of 0.05, indicating that the observed difference in accuracy between the models is statistically significant. In terms of sensitivity, the VGG-16 model performs the best, although it has a significantly lower precision score. The two-level binary classification model ranks second in sensitivity, but the key deciding factor in determining the best architecture is the F₁ score. With an impressive F₁ score of 92.1%, the two-level binary classification model demonstrates consistent performance in both the sensitivity and precision metrics. Given its near-equal accuracy to the NASNet three-class model and its superior balance of sensitivity and precision, the two-level binary classification model emerges as the ideal choice for the final layer in the application. Additionally, with a sensitivity of 91.9% on malignant cases, it demonstrates strong performance in identifying critical instances, further solidifying its suitability for deployment.

In contrast, the task of classifying relevant and irrelevant images, as shown in Table 2, involves a simpler classification challenge compared to distinguishing between malignant, benign, and normal breast tissues, where the data tend to share more overlapping features. In this case, the models exhibit near-perfect performance, as the distinguishing characteristics of irrelevant and relevant images are much more straightforward and less nuanced than those of different tissue types. Among the models tested, NASNet stands out for its superior accuracy, correctly classifying all 1043 test samples.

This outstanding performance on both datasets emphasizes the versatility of NASNet and its ability to excel in tasks of varying complexity, ranging from more challenging classifications to simpler ones. Despite the fact that this model requires more time for both training and inference compared to the other two, its ability to provide responses within milliseconds, particularly given that training occurs offline, makes it a viable choice for deployment. However, in scenarios involving online training, where quicker training and response times are critical, it would be necessary to reassess the choice of architecture and select the model best suited to meet the specific performance demands of the application.

10. Conclusions and Future Work

This paper explored the potential benefits of a hierarchical two-tier binary classification architecture compared to the traditional flat three-class classification approach for breast tumor detection in ultrasound images. Although the results show that breaking the task into two stages, first distinguishing normal tissue from tumorous tissue, and then categorizing benign and malignant tumors, did not significantly improve accuracy, the sensitivity and F₁ score metrics, combined with the hierarchical approach’s potential for improved modularity and scalability, suggest that this architecture may be more effective for the task. Among the models tested, NASNet demonstrated superior performance across almost all metrics, both in the tumor classification and the filter stage. The introduction of a filter stage, which effectively eliminates most irrelevant images before classification, plays a crucial role in enhancing the overall performance and user experience of the application developed, ensuring that only diagnostically relevant data proceeds to subsequent analysis.

Future work can build on these findings by exploring more diverse datasets and further optimizing each stage of the two-tier binary classification architecture, which is strongly believed to lead to improvements in accuracy as well as other performance metrics. Additionally, combining both architectures with other models in the same application through majority voting could enhance the final result by leveraging the strengths of each model. The results of this study provide valuable insights into the advantages of hierarchical architectures in medical image analysis, emphasizing their potential to improve diagnostic accuracy and efficiency in clinical settings, while also broadening their applicability to various medical and nonmedical imaging domains.