COVID-19 CXR Classification: Applying Domain Extension Transfer Learning and Deep Learning

Abstract

The infectious coronavirus disease-19 (COVID-19) is a viral disease that affects the lungs, which caused great havoc when the epidemic rapidly spread around the world. Polymerase chain reaction (PCR) tests are conducted to screen for COVID-19 and respond to quarantine measures. However, PCR tests take a considerable amount of time to confirm the test results. Therefore, to supplement the accuracy and quickness of a COVID-19 diagnosis, we proposed an effective deep learning methodology as a quarantine response through COVID-19 chest X-ray image classification based on domain extension transfer learning. As part of the data preprocessing, contrast limited adaptive histogram equalization was applied to chest X-ray images using Medical Information Mart for Intensive Care (MIMIC)-IV obtained from the Beth Israel Deaconess Medical Center. The classification of the COVID-19 X-ray images was conducted using a pretrained ResNet-50. We also visualized and interpreted the classification performance of the model through explainable artificial intelligence and performed statistical tests to validate the reliability of the model. The proposed method correctly classified images with 96.7% accuracy, an improvement of about 9.9% over the reference model. This study is expected to help medical staff make an integrated decision in selecting the first confirmed case and contribute to suppressing the spread of the virus in the community.

1. Introduction

Background

The infectious coronavirus disease-2019 (COVID-19) or SARS-CoV-2 virus rapidly spread and was declared a pandemic by the World Health Organization (WHO) in March 2020 [1]. As of early October 2022, the recorded number of confirmed cases worldwide was 624,142,119, and the death toll was 6,552,954 [2]. COVID-19 is a viral disease that causes severe fever, a cough, shortness of breath, sore throats, headaches, diarrhea, a loss of taste, fatigue, and a loss of smell. Severe respiratory failure can lead to death [3].

Accordingly, screening for COVID-19 confirmed cases and responding to quarantine measures are essential. The early identification of confirmed cases through a polymerase chain reaction (PCR) test and X-ray is required for screening COVID-19 confirmed cases and implementing a suitable response (quarantine, treatment, and recovery). However, it has been pointed out that the PCR test is expensive, and it takes considerable time to confirm the test results [4]. The budgeting of the expenditure resulting from the high cost of testing and the input of a professional workforce for each testing location is critical. Therefore, the economic burden of government spending is high.

It was determined that COVID-19 is a respiratory lung disease that is also one of the mutations of pneumonia. Diagnosis using a chest X-ray (CXR) for the early detection of lung disease is effective because it requires less time and money [5]. Compared to CT, X-ray imaging is cheaper, faster, and more readily accessible, and it exposes the body to far less hazardous radiation [6]. Therefore, X-ray images are an effective instrument for rapidly diagnosing pneumonia and COVID-19 in patients [7].

According to the WHO recommendation regarding chest imaging for COVID-19, chest X-ray images are recommended as complementary information for those who test negative in the initial PCR test but have suspicious COVID-19 symptoms [8]. As a result of an X-ray and CT of the patient with a false negative PCR test result, 70% of COVID-19 symptoms were confirmed by chest radiography or a chest CT [9,10]. Using X-ray images can help prevent and manage the prognosis of diseases by visualizing infected lungs along with a confirmation of lung infection [11,12].

In addition, this information can help professionals take timely decisions on admission and discharge criteria in terms of the infection stage, level of care, and patient care modalities and help to secure the need for ICU beds [3,9,13,14]. Accordingly, X-ray imaging, as an auxiliary tool, can help diagnose pneumonia and COVID-19 in patients [8,11,15]. Therefore, by using an AI model to accurately classify X-rays and enable the early screening of confirmed cases, it is possible not only to suppress the spread of disease in the community but also to reduce the cost of testing. It can also help to ensure a pre-emptive quarantine response during a pandemic that causes chaos [16,17].

Owing to the inherent qualities of machine vision, artificial intelligence (AI) and machine learning technology have achieved outstanding outcomes in the medical imaging area [18]. In an era in which COVID-19 is spreading across the world, machine learning researchers and computer scientists play a crucial role. Deep learning is one of the breakthroughs in AI. It collects the fine-grained features from the images [19]. It is a set of machine learning techniques primarily focused on automatically extracting and classifying features from images. Many challenges, including arrhythmia detection, skin cancer classification, breast cancer detection, brain disease classification, and pneumonia identification from chest X-ray images, have been successfully addressed with deep learning techniques [20]. Convolutional neural networks (CNNs) have shown promise in the analysis and classification of X-ray images using deep learning techniques [3].

Since the pandemic, research on COVID-19 disease detection and patient diagnosis by classifying images, such as X-ray, computed tomography (CT), and magnetic resonance imaging (MRI), using artificial intelligence (AI) has been actively conducted [21].

As COVID-19 spread worldwide, research was conducted to detect COVID-19 lesions on CXR images based on deep learning methodologies [16,22,23]. Owing to the severe lack of COVID-19 CXR images at the beginning of the pandemic, research predominantly focused on deep learning architectures and algorithms. To automatically identify COVID-19 lesions on CXRs, Ozturk et al. [20] proposed Dark-CovidNet to identify COVID-19 lesions in CXR. In addition, Mahmud et al. [24] proposed a CovXNet model to detect patterns of COVID-19 in the CXR images of patients. However, in these studies, the proposed models suffered from insufficient learning and testing owing to insufficient COVID-19 CXR data. These limitations significantly affected the generalization of the model performance.

In most studies using CXR images to detect and classify COVID-19 lesions, deep learning algorithms are proposed to overcome the limitations of the insufficient COVID-19 CXR image dataset, or transfer learning methodologies that are pretrained with natural images are used. However, these methodologies have to create a new algorithm when a new lung disease pandemic occurs, and in addition, a model pretrained with natural images rather than medical images has no choice but to show a low classification accuracy.

Therefore, we proposed an effective deep learning methodology for classifying COVID-19 CXR images. To effectively respond to a pandemic caused by a new lung disease such as COVID-19, it is necessary to train a model using a large dataset. However, in reality, we faced the problem of not being able to produce good results owing to the lack of COVID-19 data. To solve this, we used MIMIC-IV CXR, which contained information about 11 lung diseases, to match target domains with medical images rather than natural images to effectively classify images even without learning COVID-19 images between pretraining. We proposed a deep learning methodology based on domain extension transfer learning.

We improved the classification accuracy and generalization performance of the model using notarized MIMIC-IV data preprocessing and effective X-ray CLAHE. Based on domain extension transfer learning, the ResNet-50 model was used to effectively classify the COVID-19 CXR images through pretraining and fine-tuning. The numerical results of the classification performance of the model were visualized and interpreted using gradient-weighted class activation mapping (Grad-CAM) on the COVID-19 X-ray images containing the opinion of a radiologist. The reliability of the performance of the model was also evaluated through statistical tests.

The remainder of this paper is organized as follows. Section 2 describes the related works. Section 3 describes the dataset used in this study, the preprocessing methodology, and the deep learning methodology. Section 4 describes the hyperparameters applied to the proposed methodology, the performance evaluation indicators used in the results of the experiments, and the performance results of the final model. Section 5 presents the discussions, and Section 6 presents the conclusions and directions for future research.

2. Related Works

2.1. COVID-19 X-ray Image Classification Using Pretrained Models Based on Transfer Learning

To generalize the model performance, the classification of COVID-19 X-ray images using pretrained models based on transfer learning was performed. Apostolopoulos et al. [25] proposed a method for classifying COVID-19 CXR images using a model pretrained with natural images (ImageNet 1 k) based on the transfer learning methodology. Elpeltagy et al. [19] proposed a modified ResNet-50 pretrained model. It was trained and tested using the Mendeley dataset, which included two classes of COVID-19 and non-COVID-19 data. Owing to the five-fold cross-validation test, the average classification accuracy was 96.12%.

Sahinbas et al. [26] proposed a model using data that was directly collected from hospitals. Among the existing convolutional neural network (CNN) models, they conducted a study to detect COVID-19 lesions in X-ray images using VGG16, VGG19, ResNet, DenseNet, and Inception-V3 pretrained models. The VGG16 model showed a classification accuracy of 80%. Narin et al. [27] proposed a methodology for automatically screening COVID-19 lesions through a pretrained CNN model using 341 COVID-19 X-ray images collected by Cohen et al. as well as normal and pneumonia CXRs. The Inception-V3, ResNet-50, ResNet-101, ResNet-152, and Inception-ResNet-V2 models were used, with a five-fold cross-validation test showing that ResNet-50 achieved the highest classification performance of 99.7%.

2.2. Domain Extension Transfer Learning and Augmentations

In addition, studies have been conducted to diagnose COVID-19 using domain extension transfer learning strategies and contrast limited adaptive histogram equalization (CLAHE). Afshar et al. [28] showed that a pretrained model using the same kind of source domain (training data) as the target domain (test data) significantly improved the COVID-19 classification performance compared with applying transfer learning from ImageNet 1 k (pretrained with natural images). The experimental results demonstrated that the classification accuracy was improved by 2.5%, owing to the training and testing by matching domains with the proposed model. However, the sensitivity was 80%, showing a limitation of the study.

In addition, Basu et al. [29] matched the target and source domains with X-ray image data from scratch rather than using a pretrained model using natural images. Using AlexNet, VGGNet, and ResNet, image classification performances of approximately 82%, 90%, and 85%, respectively, were derived. In addition, Saiz et al. [30] verified that image conversion preprocessing using CLAHE has noticeable results when screening and diagnosing lung infections. The application of CLAHE resulted in a 1.6% and 9% improvement in the classification accuracy for normal and COVID-19 images in CXR image classification experiments, respectively.

As mentioned above, deep learning has been used to propose new algorithms and general transfer learning to overcome the limitations of insufficient data for COVID-19 image classification. However, effective deep learning methodologies are required to quickly and appropriately respond to COVID-19 and viruses that cause new lung diseases, such as COVID-19.

3. Materials and Methods

In this section, as shown in Figure 1, we proposed a framework that effectively classified COVID-19 CXR images through domain extension transfer learning based on ResNet-50 after applying the CLAHE technique that effectively detects lung disease lesions in MIMIC-IV CXR images.

3.1. Datasets

In this study, we used MIMIC-IV CXR data to apply domain extension transfer learning. The MIMIC-IV dataset CXR images were collected between 2011 and 2016 from intensive care patients treated at the Beth Israel Deaconess Medical Center in the United States [31]. A total of 377,110 X-ray images of healthy patients and those with various chest diseases in 12 classes were included in the dataset. The data used to classify the CXR images of COVID-19 confirmed patients were obtained from the Mendeley dataset, which comprises public data collected by El-Shafai et al. [32] in 2020. The Mendeley dataset has been cited more than 35 times in studies on COVID-19 detection published in overseas journals, with a total of 9544 X-ray images in the binary class. Each class comprises COVID-19 and non-COVID-19 images. In this study, we divided the training, validation, and testing of the Mendeley dataset into a ratio of 6:2:2 on the evaluation data of the model, as shown in

Table 1. Summary of the Mendeley dataset.

Mendeley Dataset		Training	Validation	Testing
Class	Images
COVID-19	4044	2426	809	809
Non-COVID-19	5500	3300	1100	1100
Total	9544	5726	1909	1909

.

There were many multilabel images owing to exploration data analysis (EDA) conducted on the images of the 377,110 MIMIC-IV dataset. For experimental purposes, the images were preprocessed as a dataset with only one class per image to remove inappropriate noise. Following the preprocessing procedure, we initially obtained 237,000 images. To conduct a more accurate classification of the COVID-19 X-ray images, we also classified 64,557 CXR images obtained from 237,000 images, known as posteroanterior (PA) images [33]. A PA X-ray image provides useful information for detecting lung diseases. Therefore, in this study, 237,000 images were classified as MIMIC CXR, and 64,557 images were selected as PA images (MIMIC CXR(PA)). To pretrain the model, the MIMIC CXR and MIMIC CXR(PA) datasets were divided in the ratio of 8:1:1. The MIMIC CXR(PA) dataset was chosen for the final experiment because we determined that it was the only significant dataset. The MIMIC CXR(PA) dataset comprised 51,641 training images, 6458 validation images, and 6458 test images. 11 classes were used in the experiment because the pleural other class was absent from the CXR(PA) dataset. The detailed status of the MIMIC CXR(PA) dataset is described in

Table 2. Summary of the MIMIC CXR(PA) ¹ dataset.

MIMIC CXR(PA) Dataset		Training	Validation	Testing
Class	Images
Atelectasis	2105	1684	211	211
Cardiomegaly	3106	2484	312	312
Consolidation	285	228	28	28
Edema	587	469	58	58
Fracture	643	514	64	64
Lung Lesion	810	648	81	81
Lung Opacity	4113	3290	416	416
No Finding	48,577	38,861	4857	4857
Pleural Effusion	2177	1741	217	217
Pneumonia	1676	1340	167	167
Pneumothorax	478	382	47	47
Total	64,557	51,641	6458	6458

¹ MIMIC-IV: Medical Information Mart for Intensive Care-IV; PA: posteroanterior.

.

3.2. Preprocessing

Data Augmentation

The COVID-19 CXR image data were resized to the input size of the CNN model. Rotation and horizontal flip were suitable for data augmentation during classification of the COVID-19 CXR images. This approach was effective in improving model generalization performance when applied to the classification of COVID-19 CXR images.

According to research conducted by Reza et al. [34], the main idea of CLAHE involves mapping each pixel (Equations (1) and (2)), and it is based on the image distribution of grayscale. Research has demonstrated that a medical image is produced in a decent state by balancing the pixels of the image through image conversion. The study by Reza involved two equations on CLAHE. Equation (1) represents histogram equalization, and Equation (2) represents reducing bias by using contrast limit. Explanations of these equations are as follows. M and N are the number of pixels and grayscale in each region, respectively, and $f_{i,j}\left(n\right)$ is the histogram of the $\left(i,j\right)$ region for n = 0, 1, 2 …, N − 1. An estimate of the corresponding CDF, appropriately scaled by (N − 1) for the grayscale mapping [34], is as follows:

$$f_{i,j}\left(n\right)=\frac{\left(N-1\right)}{M}\times {\displaystyle \sum }_{k=0}^n{h}_{i,j}\left(k\right) , n=1, 2, 3\dots , N-1$$

(1)

To restrict the contrast to a specific level, clip limit β is used to cut the histogram to limit the slope of Equation (1). Clip limit is related to clip factor α [34] as follows:

$$\beta =\frac{M}{N}\left(1+\frac{\alpha }{100}\left(s_{max}-1\right)\right)$$

(2)

The collected X-ray images became cloudy over time, thereby making them unsuitable for lesion detection [35]. Therefore, CLAHE was applied in the pretreatment process. The improved quality of X-ray images enhanced the classification accuracy of COVID-19 CXR images by approximately 3.9%. Figure 2 shows the results of applying CLAHE to the Mendeley dataset. Unlike hazy X-rays, X-rays with CLAHE applied to the images show a ground-glass opacity called COVID-19 lesions. In both lungs, these lesions can be identified well with the naked eye.

Table 3. Image quality comparison after CLAHE with SNR.

CLAHE
COVID-19 X-ray		Before	After	Improvement
Resized	SNR	3.9 DN	4.6 DN	0.7 DN
	SNR (dB)	1.9 dB	2.2 dB	0.3 dB

shows the results obtained through signal-to-noise ratio (SNR) indicators to indicate whether image quality improves after CLAHE in this study. Comparing the existing image and the image after CLAHE revealed that the SNR value increased in both the original and resized images. Higher SNR value means that the image quality was relatively superior. Hence, CLAHE applied in this study affected model performance improvement.

3.3. Domain Extension Transfer Learning

Pretraining and Fine-Tuning

We proposed an effective methodology for classifying COVID-19 X-ray images based on domain extension transfer learning which was conducted using a pretrained ResNet-50 model. Each pretraining process was carried out according to the target data using the MIMIC CXR and MIMIC CXR (PA) datasets.

To classify COVID-19 and non-COVID-19 CXR successfully, we matched the source domain and target domain in the medical area using a similar CXR dataset. Generally, data are limited, and it is expensive to get expert annotation in the medical area. Deep CNN training takes a long time as it needs a lot of computing and memory resources. In the absence of sufficient data, transfer learning (TL) offers a feasible option to fine-tune a CNN that has previously been pretrained on a huge collection of accessible labeled datasets from some other category. The collected COVID-19 dataset was deemed insufficient to train the CNN, which involves a sizable number of trainable parameters, from scratch to learn the complicated representative features for differentiating between COVID-19 and non-COVID-19 CXRs. Therefore, transfer learning was used to train final models for the target domain ($D_T$) and target task ($T_T$) with less annotated data by utilizing the information obtained from the source domain ($D_S$) and source task ($T_S$). The general idea of transfer learning is as follows.

A tuple of the form {$\chi$ $, P\left(X\right)$}, where $\chi$ and $P\left(X\right)$ $\left(X=x_1, x_2,\dots ,x_n \in \chi \right)$ represent the feature space and associated marginal probability distribution can be used to describe a domain $(D$). A task $T$ consists of a label space $Y$ and a conditional probability distribution $P(Y| X)$ that is typically learned from the training data, given a domain $D=\left\{\chi , P\left(X\right)\right\}.$ The idea of transfer learning is to learn the target conditional probability distribution $P\left(Y_T\right|X_T)$ in the target domains $(D_T$), using the knowledge gained from the source $(D_S$) by troubleshooting the tasks ($T_S$) [29]. Figure 3 shows the basic description of transfer learning.

The recent research has shown that transfer learning has a very limited role to play when the source and target domains are quite different in nature, such as in natural and medical images, as the networks may obtain quite distinct high-level features in the two cases. Therefore, we used each MIMIC CXR and MIMIC CXR(PA) dataset as the source data ($D_S$) to pretrain the network first, and then by using pretrained model, we applied the knowledge obtained from solving source task $\left(T_S\right)$ with source dataset to the target dataset ($D_T$), the Mendeley dataset. Therefore, we proposed a deep learning methodology that solves the target task ($T_T$) with domain extension transfer learning. Transfer learning was implemented to build up the final model to solve the target task including scarce target data ($D_T$) by utilizing the knowledge learned from the source data ($D_S$).

Fine-tuning is a deep learning technique to increase classification performance by fitting pretrained model weights to target data. Various fine-tuning strategies are also used in the field of imaging diagnostics [36]. In the biomedical field, there is a case where the classification performance was shown to improve by applying fine-tuning [37]. In this study, the ResNet-50 model was pretrained using MIMIC CXR and MIMIC CXR(PA) data based on domain extension transfer learning. Additionally, the performance of the models that were fitted to the target dataset was compared with that of ResNet-50, and the CNN models that had already been pretrained using natural images were pretrained using MIMIC CXR and MIMIC CXR(PA) data, respectively.

In this study, a fine-tuning strategy was used when the target domain dataset was insufficient and when the data in the source and target domains were similar. Here, only a few layers of the pretrained weights were fine-tuned, and the rest of the layers were frozen. As shown in Figure 4, the fine-tuning strategy was applied to the ResNet-50. The training strategy of the experiment involving fine-tuning from the 5th layer to the fully connected (FC) layer and freezing the remaining layers achieved the highest accuracy levels of COVID-19 X-ray image classification.

3.4. Classification Models

CNN Models for Image Classification

The CNN model used in this study was ResNet-50. In previous studies, ResNet-50 showed high performance in classifying COVID-19 CXR images.

ResNet was proposed by He et al. [38] in 2015. Contrary to the assumption that prediction performance improves as the neural networks of AlexNet and VGGNet deepen, performance degradation occurs owing to gradient vanishing and gradient exploding. To solve this problem, ResNet uses the skip/shortcut connection method. In the learning of the existing neural network, H(X) = X was learned, but using residual functions, H(X) = F(X) + X was learned, and residual learning was conducted so that F(X) = 0. Therefore, the problem of gradient loss was solved by suggesting a solution to approximate H(X) = X (Equations (3)–(5), Figure 4). In the formula, X represents input, Y represents output, and F(X) represents the residual function. ResNet is relatively more efficient than the existing CNN model because it uses an element-wise sum rather than the existing element-wise product. Figure 5 describes the learning method of ResNet, which is called “Residual learning”.

$$F\left(X\right):=H\left(X\right)-X$$

(3)

$$H\left(X\right)=F\left(X\right)+X$$

(4)

$$Y=F\left(X,w_i\right)+X$$

(5)

To implement the proposed methodology, each image classification model was selected as a comparison group, and a comparative experiment was conducted to determine the effectiveness of ResNet-50. When the proposed methodology was applied to ResNet-50, the highest COVID-19 CXR classification accuracy was achieved. There was a high-performance difference in classification accuracy and precision, sensitivity, specificity, and the F1-score.

Table 4. Performance of classification models after applying the proposed method on COVID-19 X-rays.

Models	Accuracy	Precision	Sensitivity	Specificity	F1-Score	Inference Time (MS)	MADDs(G)	Prams
ResNet-50	96.7%	96.2%	96.1%	97.2%	96.1%	45	4.08	23.5 M
VGG16	95.8%	95.2%	94.9%	96.5%	95.1%	225	15.46	134 M
DenseNet-121	95.6%	93.5%	96.2%	95.0%	94.8%	28	2.80	7.9 M
Inception-V3	95.9%	94.8%	95.6%	96.1%	95.2%	35	2.80	21 M
MobileNetV2	95.5%	94.3%	95.1%	95.8%	94.7%	82	0.3	3.5 M
EfficientNet-B0	96.3%	95.4%	95.9%	96.6%	95.6%	47	0.385	5.2 M

shows the results of applying the proposed methodology to each CNN model pretrained using natural images. The classification models used for the control group were VGG16 [39], DenseNet-121 [40], Inception-V3 [41], MobileNetV2 [42], and EfficientNet-B0 [43]. Figure 6 shows accuracy, inference time, MADDs, and parameters for each model.

4. Results

4.1. Experimental Settings

The experimental setting comprised the AMD Ryzen 5 5600X 6-core processor (CPU), 32 GB RAM, GeForce 3060TI (NVIDIA), Python 3.7.11, and Pytorch 1.11.0. Cross-entropy was used as the loss function, and AdamW was used as the optimizer [44]. The optimal hyperparameters for the suggested methodology were determined by analyzing the training strategy for each stage using the pretraining and fine-tuning the settings of ConvNet, which shows a high performance in image classification [45]. To determine the optimal value, we used the cosine decay learning rate schedule. The hyperparameters used in the experiment were equally applied to all CNN model comparison groups, as listed in

Table 5. Optimal hyperparameter values.

Configurations	Pretraining	Fine-Tuning
Input Size	2242	2242
Optimizer	AdamW
Learning Rate	1 × 10−2	1 × 10−5
Weight Decay	0.05	1 × 10−8
Optimizer Momentum	$\beta 1$, $\beta 2=$ 0.9, 0.999
Batch Size	256
Training Epochs	100	20
Learning Rate Schedule	Cosine Decay
Warm-up Epochs	10	N/A
Warm-up Schedule	Linear	N/A
CLAHE	Clip Limit = 8, Grid Size = 15

.

4.2. Evaluation

We used a confusion matrix to evaluate the image classification performance (

Table 6. Confusion matrix of binary classification.

		Predicted Class
		COVID-19	Non-COVID-19	Total
Actual Class	COVID-19	$N_{11}$	$N_{12}$	${\displaystyle \sum _{i=1}^2}{N}_{1i}$
	Non-COVID-19	$N_{21}$	$N_{22}$	${\displaystyle \sum _{i=1}^2}{N}_{2i}$
	Total	${\displaystyle \sum _{i=1}^2}{N}_{1i}$	${\displaystyle \sum _{i=1}^2}{N}_{2i}$	$N$

). As evaluation indicators, accuracy, precision, sensitivity, specificity, F1-score (harmonic mean), Matthews correlation coefficient (MCC), and balanced accuracy were used, and each value was calculated as shown in Equations (6)–(12).

$$\mathrm{Accuracy}=\frac{1}{N}{{\displaystyle \sum }}_{i=1}^2{N}_{ii}/\frac{\left(TP+TN\right)}{\left(TP+FP+FN+TN\right)}$$

(6)

$$\mathrm{Precision}=\frac{N_{jj}}{{{\displaystyle \sum }}_{i=1}^2{N}_{ij}}, \mathrm{where} \mathrm{j}=1/\frac{TP}{\left(TP+FP\right)}$$

(7)

$$\mathrm{Sensitivity}=\frac{N_{jj}}{{{\displaystyle \sum }}_{i=1}^2{N}_{ji}}, \mathrm{where} \mathrm{j}=1/\frac{TP}{\left(TP+FN\right)}$$

(8)

$$\mathrm{Specificity}=\frac{N_{jj}}{{{\displaystyle \sum }}_{i=1}^2{N}_{ji}}, \mathrm{where} \mathrm{j}=2/\frac{TN}{\left(FP+TN\right)}$$

(9)

$$\mathrm{F}1\text{-}\mathrm{score}=\frac{2\times \left(Precisio{n}_i\times Sensitivit{y}_i\right)}{Precisio{n}_i+Sensitivit{y}_i}$$

(10)

$$\mathrm{MCC}=\frac{\left(TP\times TN\right)-\left(FN\times FP\right)}{\sqrt{\left(TP+FP\right)\times \left(TP+FN\right)\times \left(TN+FP\right)\times \left(TN+FN\right)}}$$

(11)

$$\mathrm{Balanced} \mathrm{accuracy}=\frac{Sensitivity+Specificity}{2}$$

(12)

4.3. Evaluation Results

Table 7. Performance of the baseline and proposed models.

Models	Accuracy $(\pm \mathit{s}\mathit{t}\mathit{d})$	Precision $(\pm \mathit{s}\mathit{t}\mathit{d})$	Sensitivity $(\pm \mathit{s}\mathit{t}\mathit{d})$	Specificity $(\pm \mathit{s}\mathit{t}\mathit{d})$	F1-Score $(\pm \mathit{s}\mathit{t}\mathit{d})$	MCC $(\pm \mathit{s}\mathit{t}\mathit{d})$	Balanced Accuracy $(\pm \mathit{s}\mathit{t}\mathit{d})$
Baseline Model	86.8% ($\pm 0.005$)	83.1% ($\pm 0.058$)	86.5% ($\pm 0.085$)	87.0% ($\pm 0.059$)	84.2% ($\pm 0.047$)	73.4% ($\pm 0.004$)	86.7% ($\pm 0.001$)
Proposed Model	96.7% ($\pm 0.025$)	96.2% ($\pm 0.016$)	96.1% ($\pm 0.018$)	97.2% ($\pm 0.092$)	96.1% ($\pm 0.097$)	93.2% ($\pm 0.003$)	96.6% ($\pm 0.008$)
Improvement Ratio	9.9%	13.1%	9.6%	10.2%	11.9%	20%	9.9%

shows the results of calculating the performance of the model to which the proposed methodology was applied using the performance evaluation indicators. Using the methodology proposed, a classification accuracy level of 96.7%, a precision level of 96.2%, a sensitivity level of 96.1%, a specificity level of 97.2%, an F1-score of 96.1%, an MCC level of 0.93, and a balanced accuracy level of 96.6% were obtained. The proposed model improved accuracy by 9.9%, precision by 13.1%, sensitivity by 9.6%, specificity by 10.2%, the F1-score by 11.9%, MCC by 0.2, and balance accuracy by 9.9% in each part. Compared with the baseline model, the classification performance of the model improved when the proposed methodology was applied. It was confirmed that the performance was significantly improved in sensitivity and specificity, which are important performance measures in the biomedical field.

Figure 7 shows a confusion matrix for the actual and predicted values of the COVID-19 CXR binary classification. Figure 7a shows the classification result of the baseline model, and Figure 7b shows the classification result of the model to which the proposed methodology was applied. In the confusion matrix, the horizontal axis represents the predicted label value, and the vertical axis represents the actual label value. The values from the left-hand side to the right-hand side are from the good prediction of the actual label of the model. The baseline model showed a low accuracy for COVID-19 and non-COVID-19 images. However, the model to which the proposed methodology was applied showed a significant increase in the predicted label value of the actual label value. It can also be observed that the classification accuracy improved in all actual classes.

Table 8. Results of comparative experiments.

Models	Accuracy $(\pm \mathit{s}\mathit{t}\mathit{d})$	Precision $(\pm \mathit{s}\mathit{t}\mathit{d})$	Sensitivity $(\pm \mathit{s}\mathit{t}\mathit{d})$	Specificity $(\pm \mathit{s}\mathit{t}\mathit{d})$	F1-Score $(\pm \mathit{s}\mathit{t}\mathit{d})$	MCC $(\pm \mathit{s}\mathit{t}\mathit{d})$	Balanced Accuracy $(\pm \mathit{s}\mathit{t}\mathit{d})$
Baseline (Non-pretrained/Mendeley)	86.8% ($\pm 0.005$)	83.1% ($\pm 0.058$)	86.5% ($\pm 0.085$)	87.0% ($\pm 0.059$)	84.2% ($\pm 0.047$)	73.4% ($\pm 0.004$)	86.7% ($\pm 0.001$)
Proposed (Pretrained/MIMIC(PA)—Mendeley)	96.7% ($\pm 0.025$)	96.2% ($\pm 0.016$)	96.1% ($\pm 0.018$)	97.2% ($\pm 0.092$)	96.1% ($\pm 0.097$)	93.2% ($\pm 0.003$)	96.6% ($\pm 0.008$)
Model 1 (Pretrained/Mendeley)	95.8% ($\pm 0.061$)	94.8% ($\pm 0.023$)	95.4% ($\pm 0.024$)	96.1% ($\pm 0.013$)	95.1% ($\pm 0.104$)	91.3% ($\pm 0.005$)	95.8% ($\pm 0.013$)
Model 2 (Non-pretrained/MIMIC–Mendeley)	95.6% ($\pm 0.027$)	94.4% ($\pm 0.014$)	95.3% ($\pm 0.023$)	95.9% ($\pm 0.037$)	94.8% ($\pm 0.013$)	90.1% ($\pm 0.002$)	95.6% ($\pm 0.018$)
Model 3 (Non-pretrained/MIMIC(PA)–Mendeley)	95.6% ($\pm 0.023$)	94.5% ($\pm 0.022$)	95.0% ($\pm 0.018$)	96.0% ($\pm 0.026$)	94.8% ($\pm 0.019$)	90.1% ($\pm 0.003$)	95.5% ($\pm 0.014$)
Model 4 (Pretrained/MIMIC–Mendeley)	95.9% ($\pm 0.032$)	95.8% ($\pm 0.023$)	94.5% ($\pm 0.005$)	97.0% ($\pm 0.005$)	95.2% ($\pm 0.005$)	91.2% ($\pm 0.002$)	95.7% ($\pm 0.012$)

shows the results of the comparative experiment conducted by applying the MIMIC CXR and MIMIC CXR (PA) data to ResNet-50. The baseline model results are from the image classification by training and testing the ResNet-50, using Mendeley datasets. The model to which the proposed methodology was applied used the ResNet-50, which was pretrained using natural images and pretrained using the MIMIC CXR (PA) data based on domain extension transfer learning, after which fine-tuning was performed using the Mendeley dataset. Model 1 is the result of fitting the ResNet-50 model pretrained using natural images to the Mendeley dataset. Models 2 and 3 are the results of applying domain extension transfer learning to the Mendeley datasets through ResNet-50 models pretrained using MIMIC CXR and MIMIC CXR (PA) data, respectively. Finally, Model 4 is the result of the proposed methodology using the MIMIC CXR dataset.

The experimental results demonstrate that the PA image datasets significantly improve the classification performance of CXR images and that the use of pretrained models is more effective in classifying images. Additionally, the proposed methodology for pretraining and fine-tuning the target domain rather than simply using a pretrained model is most effective in classifying COVID-19 CXR images.

Figure 8 shows the training and validation accuracies and losses during the training.

Table 9. Performance comparison of the proposed methodology with other studies.

Author	Dataset	DL Model	All Data	COVID Data	Train All/COVID	Validation All/COVID	Test All/COVID	Sensitivity	Specificity
Proposed	COVID-19 and non-COVID-19	ResNet-50	74, 101 (MIMIC and Mendeley)	5500	54,941/2426	7558/809	7558/809	96.1%	97.2%
[19]		ResNet-50	9545 (Mendeley)	5500	7636/4400	-/-	1909/1100	97.7%	94.9%
[20]		Dark-CovidNet	627	127	400/101	100/26	-/-	95.1%	95.3%
[24]		CovidXNet	610	305	244/244	61/61	-/-	97.8%	94.7%
[18]		RadiomiX	1381	181	1104/145	-/-	276/36	78.9%	91.0%
[46]		DenseNet-121	2724	1029	1059/526	328/177	1337/326	84.0%	93.0%
[47]		Inception-ResNet-v2	905	419	534/242	92/43	279/134	82.8%	84.3%
[48]		U-Net	5212	275	3285/657	597/120	1330/266	96.3%	93.6%

shows the performance comparison of the proposed methodology with other studies. Compared to previous studies, our study showed significant performance improvements in sensitivity and specificity indicators when classifying COVID-19 and non-COVID-19 images. Compared to Elpeltagy at el. [19] who conducted experiments on the same dataset as ours, the model proposed in this study showed significant performance differences in precision and specificity before the five-fold test. After a five-fold test measuring the stability of the model within a small data set, the models showed significant performance differences in accuracy, precision, specificity, and the F1-score. The results of applying the proposed methodology to COVID-19 and non-COVID-19 X-ray imaging classification studies were effectively confirmed.

4.4. Statistical Test

In this study, the difference between the performance levels of the two models was verified through statistical tests for the model to which the baseline model and the proposed methodology were applied. The two-sample test was conducted, and the results are listed in

Table 10. Two sample t-test on the study results.

t-Test
Hypothesis	p-Value	95% c. i
$H_0: {\mu }_{Base}={\mu }_{Proposed,Model 1 to 4}$	p < 0.05	[9.879, 9.898] [8.962, 8.981] [8.816, 8.835] [8.809, 8.828] [9.132, 9.151]
$H_1: {\mu }_{Base} \ne {\mu }_{Proposed,Model 1 to 4}$

. It was confirmed that the difference in the image classification performance between the baseline model and the model to which the proposed methodology was applied was statistically significant because the p-value was less than 0.05, and the 95% confidence interval did not include 0. We also confirmed that the difference in the classification performance between the baseline model and each model from model 1 to 4 was statistically significant because the p-value was less than 0.05, and the 95% confidence interval did not include 0.

4.5. XAI

XAI addresses the problem of CNN models where they numerically show good classification performance but cannot intuitively interpret the results. We used Grad-CAM (a slope-based class activation map) among the XAI methodologies [49]. The Grad-CAM is an XAI methodology that follows CAM [50], and a generalized version of CAM utilizes information that influences the class prediction to demonstrate high performance in the artificial neural networks that handle diverse and complex tasks. In this study, the X-ray image diagnosed by the Korean Society of Radiology was used to visualize and interpret the image classification results of the Mendeley dataset for the model to which the proposed methodology was applied. The results of visualization and interpretation validated the classification performance of the model.

Figure 9 shows the results of the visualization through Grad-CAM in the COVID-19 and normal CXRs, which include expert opinions from the Korean Society of Thoracic Radiology (KSTR) [51] on the classification performance of the model to which the proposed methodology was applied and that of the baseline model. The non-COVID-19 CXR represents a CXR image of a normal lung of a male in his 50s, and the model to which the proposed methodology was applied clearly shows that there is no ground-glass opacity related to COVID-19 around the lungs. The COVID-19 X-ray represents a CXR diagnosed by a specialist showing that ground-glass opacity is spread over the entire left and right lungs of a woman in her 50s. The results visualized by the model are the same as those diagnosed by domain experts. The baseline model provided an interpretation with a slightly larger error than the model to which the proposed methodology was applied.

The Mendeley data provided the lesion locations within the image. As a result of applying Grad-CAM using the information, the location of the lesion was detected from the Mendeley data. The proposed model confirmed the superiority of the classification model by accurately visualizing lung lesions showing symptoms of COVID-19 with a high probability value of over 96.1% compared to the reference model.

5. Discussion

We proposed a deep learning methodology that can be used to effectively classify COVID-19 CXR images. The MIMIC-IV CXR and Mendeley datasets were used, and effective CLAHE was applied for detecting COVID-19 X-ray lesions. The classification model demonstrated improvements in the accuracy and generalization performance of the classification of COVID-19 CXR images through domain extension transfer learning based on ResNet-50. The classification model showed high-performance levels, with an accuracy of 96.7%, precision level of 96.2%, sensitivity level of 96.1%, specificity level of 97.2%, and an F1-score of 96.1%. As a result of applying domain extension transfer learning and the ImageNet pretrained ResNet-50 model, the performance of the classification model in terms of accuracy improved by 9.9%, precision improved by 13.1%, sensitivity by 9.6%, specificity by 10.2%, and the F1-score by 11.9%, when domain extension transfer learning was applied on the MIMIC CXR(PA) data.

To show that the proposed methodology is effective, we carried out comparisons of it with six state-of-the-art studies: Dark-CovidNet [20], CovidXNet [24], RadiomiX [18], DenseNet-121 [46], Inception-ResNet-v2 [47], and U-Net [48]. We also compared the proposed methodology with the study with the same dataset [19]. Among all the methods, the proposed methodology showed the highest performance of 97.2% in specificity, which is one of the important criteria in biomedical image classification.

Using Grad-CAM, one of the XAI methodologies, the numerical results of the proposed methodology were visualized to interpret the results of the performance of the model. A COVID-19 CXR image obtained from the Korean Society of Thoracic Radiology was cited and applied to Grad-CAM. Grad-CAM characterized the lung lesions, and the radiologist findings were used to confirm the superiority of the classification model. Furthermore, by verifying the reliability of the model performance through statistical tests, we showed that the proposed methodology can be used to effectively classify COVID-19 CXR images.

The results of applying the proposed methodology for matching the source domain and the target domain are as follows. The pretraining was conducted with the MIMIC-IV dataset without COVID-19 X-ray images. A fine-tuning strategy was performed to classify the COVID-19 images and non-COVID-19 images well. It was confirmed that the proposed methodology is effective through model performance figures and visual results.

We believe that if another new pandemic with similar lung diseases breaks out, we could apply the proposed methodology to a new scarce CXR image dataset. Therefore, we could quickly respond to screening for new viruses and prevent the spreading of viruses in the local society. In addition, by applying the proposed methodology not only would a quick quarantine response be possible but also a reduction in the burden of a large expenditure to support professionals, expensive inspection kits, and inspection sites. Moreover, the proposed methodology is able to provide visualized information to medical staff and professionals to help them to make good decisions regarding the level of treatment. Therefore, through the proposed methodology, we can build up a proper medical system for patients and the local society.

Recently, COVID-19 lesion detection and classification studies have been conducted using CT and MRI images as well as X-rays. The related studies have achieved more than 95% of the state-of-the-art. In future research, we will apply the domain extension transfer learning presented in this study to the areas of CT and MRI to confirm its applicability in the overall medical image classification field.

6. Conclusions

In this study, we proposed a model for classifying COVID-19 CXR images through domain extension transfer learning. It was confirmed that our proposed model was the most effective in classifying COVID-19 CXR images by performing pretraining and fine-tuning strategies on the target domain. However, there were the following limitations in carrying out the study:

• The proposed methodology has limitations in that it is applied within the scope of CXR. By additionally collecting CT and MRI medical images and applying them to the proposed model, we are planning an integrated multimodal study by securing patient information related to COVID-19 symptoms and the diagnostic information of medical staff.
• To prevent the increase in asymptomatic infections due to the spread of vaccines, we plan to expand the scope of the research by additionally collecting the biometric information of asymptomatic patients.
• In addition, the proposed methodology has limitations in the scope of research applied only to the COVID-19 dataset. In the future, we plan to collect various new lung disease data and apply them to the proposed model for testing.

This study confirms that artificial intelligence significantly contributes to the achievement of pre-emptive and appropriate quarantine responses to COVID-19. It is expected to help reduce the social and economic burden caused by the pandemic and establish a medical system.