4.1. Performance Indicators
Our research, which involved both traditional machine learning and fine-tuned deep learning approaches, demonstrates that fine-tuning deep learning models can significantly enhance predictability and accuracy. The results reveal varied performance among different classifiers, reflecting their distinct strengths and weaknesses.
As shown in
Table 10. The CNN achieved an accuracy of 82%, while SVM reached 83%. These results indicate that traditional machine learning methods, while effective, often fall short compared with advanced deep learning techniques. The Random Forest Classifier recorded an accuracy of 81%, which is slightly lower, suggesting that its ensemble approach may not capture the complex patterns in skin cancer data as effectively as deep learning models as shown in
Table 10. EfficientNetB0, a fine-tuned deep learning model, achieved an accuracy of 84% as shown in
Table 10. This improvement over CNN and SVM highlights EfficientNetB0’s capability to leverage efficient model architecture for better performance in complex classification tasks. ResNet34 also performed well with an accuracy of 82% as shown in
Table 10, showing that while residual networks offer significant advantages, their performance can be highly dependent on specific tuning parameters and dataset characteristics. VGG16 showed notable performance with an accuracy of 85% as shown in
Table 10, demonstrating the efficacy of its deep network architecture in learning intricate features from the data. Inception_v3 and DenseNet121 achieved accuracies of 81% and 87%, respectively as shown in
Table 10. DenseNet121’s superior accuracy suggests that its dense connectivity pattern effectively facilitates feature reuse and gradient flow, leading to enhanced model performance.
For all fine-tuned models, we introduced a flatten layer followed by three dense layers. The flatten layer transforms multidimensional feature maps into a one-dimensional vector, ensuring efficient data processing. The dense layers refine feature representation, capturing intricate patterns and relationships within the data. The use of LeakyReLU activation functions in the dense layers addresses the vanishing gradient problem and ensures stable training. Finally, the output dense layer with a sigmoid activation function simplifies binary classification, providing clear decision boundaries for healthcare professionals. The incorporation of these additional layers improves the efficiency and performance of the neural network models, leading to enhanced accuracy and reliability in skin cancer classification. The comparative performance of the classifiers underscores the importance of model architecture and tuning in achieving optimal results.
Precision is a metric that evaluates the accuracy of predictions by measuring the proportion of correctly classified positive instances out of all predicted positive instances. It is computed by dividing the number of true positive predictions by the sum of true positive and false positive predictions. A high precision value indicates a well-performed model. Precision is defined as follows:
where
TP is true positive and
FP is false positive.
Recall is the ratio of all positively classified classes with a positive outcome that is correctly predicted. A good model should have a high recall rate. Recall is defined as follows:
where
FN is a false negative. A high F1-score indicates high precision and recall because the score contains information about these two variables. It is defined as follows:
The ROC curve is a graphical representation of a model’s ability to distinguish between different classes. It plots the True Positive Rate (Sensitivity) against the False Positive Rate (1 − Specificity) at various threshold settings. A perfect model would reach the top-left corner, indicating a high True Positive Rate and a low False Positive Rate.
against
where the following definitions are used:
AUC is the area under the ROC curve and provides a single metric to summarize the model’s performance. The AUC value ranges from 0 to 1, where 1 indicates a perfect model, 0.5 suggests a model with no discriminative ability (equivalent to random guessing), and 0 indicates a completely wrong model.
where the following definitions are used:
The Precision–Recall curve is a plot of Precision (Positive Predictive Value) against Recall (Sensitivity) for different threshold values. Precision measures the proportion of true positive predictions among all positive predictions, while Recall measures the proportion of true positives among all actual positives. This curve is especially useful when dealing with imbalanced datasets, where one class is much more common than the other.
CNN: Benign: 141/186 (76%), Malignant: 122/144 (85%) SVM: Benign: 152/186 (82%), Malignant: 123/144 (85%) Random Forest Classifier:Benign: 151/186 (81%), Malignant: 119/144 (83%) Fine-tuned VGG-16:Benign: 153/186 (82%), Malignant: 133/144 (92%) Fine-tuned ResNet-34: Benign: 159/186 (86%), Malignant: 101/144 (70%) Fine-tuned DenseNet-121: Benign: 163/186 (88%), Malignant: 122/144 (85%) Fine-tuned Inception-v3: Benign: 161/186 (87%), Malignant: 108/144 (75%) Fine-tuned EfficientNetB0: Benign: 155/186 (83%), Malignant: 123/144 (85%)
The fine-tuned DenseNet-121 achieved the highest correct classification rate for benign lesions (88%) and performed well on malignant lesions (85%). The fine-tuned VGG-16 also showed strong performance, with 82% for benign lesions and 92% for malignant lesions. ResNet-34, while achieving the highest accuracy for benign lesions (86%), had the lowest performance for malignant lesions (70%). The other models demonstrated competitive performance, with varying strengths in benign and malignant classifications.
Fine-tuned DenseNet-121 has the highest AUC of 0.87, indicating the best overall performance in distinguishing between benign and malignant lesions. Fine-tuned VGG-16 follows closely with an AUC of 0.86, reflecting strong classification ability. SVM and EfficientNetB0 also show strong performance with AUCs of 0.84. ResNet-34 has the lowest AUC of 0.78, suggesting it is less effective in distinguishing between the classes compared with other models, as shown in
Figure 12,
Figure 13,
Figure 14,
Figure 15,
Figure 16,
Figure 17,
Figure 18 and
Figure 19.
Fine-tuned VGG-16 and DenseNet-121 exhibit strong performance with high precision and recall, making them well suited for applications requiring reliable positive classification. Models like SVM and EfficientNetB0 also show high precision and recall but are slightly lower compared with VGG-16 and DenseNet-121, as shown in
Figure 20,
Figure 21,
Figure 22,
Figure 23,
Figure 24,
Figure 25,
Figure 26 and
Figure 27.
4.3. Statistical Significance
The statistical significance to be used in this study are mean Accuracy, Standard Deviation Accuracy, and Confidence Interval.
Mean accuracy is the average performance of the model over multiple tests. If one tests one’s model several times, mean accuracy tells one, on average, how often the model correctly predicts the outcome.
where the following definitions are used:
N is the number of tests or experiments.
is the accuracy obtained in the i-th test.
averages the sum of accuracies to provide the mean.
Confidence interval gives one a range of values within which one can be fairly sure the true accuracy of the model lies. It is like saying we are 95% confident that the model’s true accuracy is between these two numbers.
where the following definitions are used:
is the average accuracy calculated previously.
t is the t-score (or z-score) for the desired confidence level (e.g., 1.96 for a 95% confidence level).
is the measure of variability in accuracy.
is the square root of the number of tests, scaling the standard deviation to reflect sample size.
Standard deviation of accuracy measures how much the accuracy varies across different tests. It tells one whether the model’s performance is consistent or if it fluctuates a lot.
where the following definitions are used:
is the average accuracy calculated previously.
N is the number of tests or experiments.
is the accuracy obtained in the i-th test.
The term normalizes the sum of squared deviations to account for sample size.
Fine-tuned VGG16: The mean accuracy of 0.87 indicates that, on average, the model correctly classifies 87% of the test samples. The standard deviation of 0.01 reflects very little variability in its performance across different experiments. The 95% confidence interval, ranging from 0.86 to 0.87, suggests that if we repeated the experiments multiple times, the true accuracy would fall within this range 95% of the time, as shown in
Table 11.
Fine-tuned ResNET-34: With a mean accuracy of 0.79, RESNET correctly identifies 79% of the test samples on average. The standard deviation of 0.02 shows slightly more variation in performance across tests. The confidence interval between 0.78 and 0.80 indicates that we can be 95% confident the true accuracy lies within this narrow range, as shown in
Table 11.
Fine-tuned DenseNET-121: This model also achieves a mean accuracy of 0.87, meaning it performs similarly to VGG16 by correctly classifying 87% of the samples on average. The standard deviation of 0.02 suggests some minor variability. The 95% confidence interval from 0.86 to 0.87 confirms consistent performance, with the true accuracy likely falling within this range, as shown in
Table 11.
Fine-tuned Inception_v3: With a mean accuracy of 0.82, this model correctly classifies 82% of the samples on average. The standard deviation of 0.03 indicates greater variability in results. The confidence interval of 0.81 to 0.83 suggests a moderate range where the true accuracy is likely to be found, as shown in
Table 11.
Fine-tuned EfficientNETB0: Achieving a mean accuracy of 0.85, this model correctly identifies 85% of the samples on average. The standard deviation of 0.012 indicates very consistent performance across different tests. The confidence interval, from 0.84 to 0.85, suggests that the true accuracy is tightly confined within this range as shown in
Table 11.
CNN: With a mean accuracy of 0.83, the CNN correctly classifies 83% of the samples on average. The standard deviation of 0.03 indicates some variability. The confidence interval from 0.85 to 0.87 suggests a range in which the true accuracy likely lies, though this range is broader than some other models, as shown in
Table 11.
SVM: This model achieves a mean accuracy of 0.83, correctly identifying 83% of the samples on average. The standard deviation of 0.02 suggests minimal variability. The confidence interval from 0.83 to 0.84 indicates that the true accuracy is likely to fall within this narrow range, as shown in
Table 11.
Random Forest Classifier: With a mean accuracy of 0.82, RFC correctly classifies 82% of the test samples on average. The standard deviation of 0.02 shows consistent performance. The confidence interval between 0.81 and 0.82 suggests that the true accuracy is within this range with 95% confidence, as shown in
Table 11.