Enhancing Acute Bilirubin Encephalopathy Diagnosis with Multi-Modal MRI: A Deep Learning Approach

Abstract

Background: Acute Bilirubin Encephalopathy (ABE) is a major cause of infant mortality and disability, making early detection and treatment essential to prevent further progression and complications. Methods: To enhance the diagnostic capabilities of multi-modal Magnetic Resonance Imaging (MRI) for ABE, we proposed a deep learning model integrating an attention module (AM) with a central network (CentralNet). This model was tested on MRI data from 145 newborns diagnosed with ABE and 140 non-ABE newborns, utilizing both T1-weighted and T2-weighted images. Results: The findings indicated the following: (1) In single-modality experiments, the inclusion of AM significantly improved all the performance metrics compared to the models without AM. Specifically, for T1-weighted MRI, the accuracy was 0.639 ± 0.04, AUC was 0.682 ± 0.037, and sensitivity was 0.688 ± 0.09. For the T2-weighted images, the accuracy was 0.738 ± 0.039 and the AUC was 0.796 ± 0.025. (2) In multi-modal experiments, using T1 + T2 images, our model achieved the best accuracy of 0.845 ± 0.018, AUC of 0.913 ± 0.02, and sensitivity of 0.954 ± 0.069, compared to models without an AM and CentralNet. The specificity remained relatively stable, while the precision and F1 scores significantly increased, reaching 0.792 ± 0.048 and 0.862 ± 0.017, respectively. Conclusions: This study emphasizes the effectiveness of combining attention modules with CentralNet, significantly enhancing the accuracy of multi-modal MRI in classifying ABE. It presents a new perspective and possibility for the clinical application of multi-modal MRI imaging in the diagnosis of ABE.

1. Introduction

1.1. Background

Neonatal jaundice, also known as neonatal hyperbilirubinemia, is a common condition in newborns, manifested by yellowing of the skin and eyes [1,2]. This condition arises from the accumulation of bilirubin, a yellow pigment produced during the normal breakdown of red blood cells. Newly formed bilirubin, being lipophilic and non-water-soluble, is referred to as unconjugated bilirubin. In the bloodstream, it predominantly binds to albumin for safe transport to the liver. Once in the liver, unconjugated bilirubin is conjugated with glucuronic acid, forming conjugated bilirubin, which is water-soluble and more easily excreted via bile [3]. While most full-term infants experience physiological jaundice, typically benign and self-resolving within a few weeks or treatable through short-term phototherapy, some newborns may struggle to efficiently process unconjugated bilirubin due to an immature hepatic enzyme system, leading to more pronounced jaundice. Additionally, certain diseases and conditions can disrupt this metabolic process, resulting in elevated bilirubin levels. In some infants, rapidly increasing bilirubin levels pose a risk for developing kernicterus. If bilirubin crosses the blood–brain barrier, it can lead to the death of brain cells, resulting in Acute Bilirubin Encephalopathy (ABE) [4,5]. Without prompt intervention, ABE can progress to kernicterus, a severe neurological condition causing permanent brain damage, lifelong disabilities such as cerebral palsy, hearing loss, and developmental problems, or even death [6,7]. Therefore, the early identification and accurate diagnosis of ABE are crucial in neonatal care to prevent these severe outcomes.

Traditional diagnostic methodologies for ABE predominantly rely on clinical assessments and serum bilirubin measurements. Clinicians typically initiate the diagnosis of Neonatal Acute Bilirubin Encephalopathy (NABE) by noting signs of jaundice, such as the yellowing of skin and eyes. These clinical observations are commonly verified through blood tests that measure the total serum bilirubin (TSB) or transcutaneous bilirubin (TcB) levels. However, the reliance on these traditional diagnostic methods presents significant limitations [8]. The visibility of jaundice, crucial for initial assessment, can be markedly less apparent in darker-skinned infants, potentially leading to underdiagnosis or delayed intervention. Additionally, the variability in serum bilirubin, affected by age, health status, and blood test timing, can introduce diagnostic uncertainties, further complicating the effective identification and management of NABE.

The Bilirubin-Induced Neurologic Dysfunction (BIND) scoring system is commonly used in conjunction with traditional methods to enhance accuracy in assessing neurological impairments. The BIND score provides an objective tool by combining clinical features, abnormalities in brain Magnetic Resonance Imaging (MRI), and aberrant auditory brainstem responses [9]. This system categorizes impairment severity into mild (1–3 points), moderate (4–6 points), and severe (7–9 points) levels based on parameters like mental status, muscle tone, and crying characteristics [10]. A score of 0 indicates normal neurological function in newborns. Despite its significance in evaluating NABE and identifying bilirubin deposition within brain nuclei, the BIND scoring system has limitations. Neurological assessments in suspected ABE cases often focus on symptoms of central nervous system involvement, such as lethargy, hypotonia, irritability, high-pitched crying, poor feeding, and, in severe cases, seizures. However, these symptoms are not exclusive to ABE and may overlap with other neonatal diseases, making differential diagnosis challenging without more definitive tests. Moreover, there is a lack of sensitivity in detecting early brain changes caused by bilirubin, often leading to severe damage before clinical symptoms manifest. Therefore, the need for more advanced and sensitive diagnostic tools is highlighted, especially in scenarios where direct observation of bilirubin deposition in the brain is challenging.

1.2. Related Works

In recent years, brain MRI has emerged as a vital tool for diagnosing neurological conditions, including ABE [11,12]. MRI provides detailed images of the brain, enabling clinicians to detect subtle changes and plan appropriate treatment. Research has particularly focused on specific MRI techniques such as T1, T2, and Apparent Diffusion Coefficient (ADC) imaging in patients with ABE. T1-weighted imaging often shows increased signal intensity in the bilateral Globus Pallidus (GP) during the acute phase of ABE, attributed to the accumulation of bilirubin in these areas [13,14,15]. However, these findings are not consistent across all patients, as some individuals with elevated bilirubin levels do not show these changes. T2-weighted imaging abnormalities are less common during the acute phase of ABE but may be observed in more severe or advanced cases [16]. These changes can include hyperintensity in the bilateral GP, which may correspond with T1 hyperintensity seen during the acute phase. Diffusion-weighted imaging (DWI) and its derived ADC values have shown promise in correlating with blood bilirubin levels, although their use in routine diagnosis is still being explored [17,18]. Research findings indicate that Magnetic Resonance Spectroscopy (MRS) plays a significant role in the differential diagnosis of Neonatal Bilirubin Encephalopathy (NBE) and severe hyperbilirubinemia, especially when NBE presents with atypical (subtle) symptoms and conventional MRI fails to reveal distinct abnormalities [19,20].

The advent of advanced medical Image analysis techniques, particularly in the realms of traditional machine learning and deep learning, has revolutionized the approach to clinical disease diagnosis. Liu et al. used machine learning techniques to differentiate between abnormal brain enhancement and normal myelination in neonates [21]. Their method involved manual segmentation of the region of interest (ROI) on MRI images, feature extraction, and the selection of specific features based on T1-weighted images. This approach, while effective, is labor-intensive and not easily adaptable to clinical settings due to the need for manual segmentation and feature definition. However, Wu et al. explored the use of deep learning networks, specifically ResNet18 and DenseNet201, for the classification of multi-modal MRI images in ABE [22]. Their findings suggested that the integration of data from multiple imaging modalities could significantly improve the classification performance of ABE. However, a limitation of this approach is that simple concatenation of data from different modalities might not fully capture the unique characteristics of each, potentially limiting the neural network’s performance. The study by Zhang et al. [23]. presents a novel approach for differentiating ABE from normal myelination in newborns. They developed a feature fusion attention network, utilizing an attention mask and element-wise multiplication for feature fusion. Their experimental results demonstrated the feasibility of using attention modules and feature fusion to enhance the classification performance of ABE. However, the classification outcomes were not ideal due to certain limitations in the model structure. This work underscores the potential and challenges of applying deep learning techniques in medical image analysis for ABE classification.

1.3. Contributions of This Study

In this study, we have developed a new multi-modal MRI classification network specifically designed for diagnosing Acute Bilirubin Encephalopathy (ABE), which represents a significant step forward in medical imaging and diagnostic procedures. The key scientific contributions of our research are as follows:

Effective integration of attention mechanisms: Our model, built upon the 3D ResNet18 architecture, innovatively incorporates spatial and channel attention modules within the feature extraction networks for T1- and T2-weighted MRI images. This integration allows the network to focus precisely on crucial features within specific areas and channels of the images, thereby significantly enhancing the accuracy of ABE diagnosis. This demonstrates a marked improvement over traditional methods, showcasing the power of attention mechanisms in medical image analysis.

CentralNet as a key feature fusion method: A central element of our architecture is CentralNet, strategically positioned between the feature extraction models of T1 and T2 modal images [24]. CentralNet achieves effective feature fusion by calculating the weighted sum of different modal feature data and their preceding layers. This novel approach fosters mutual supervision and learning in the cross-modal feature extraction process, enhancing the model’s precision in classifying ABE. The successful integration of CentralNet and its validation through several control experiments underline its efficacy, paving the way for new possibilities in the clinical application of multi-modal MRI imaging for diagnosing ABE.

Collectively, these contributions highlight the effectiveness of our proposed model in improving the diagnostic accuracy of multi-modal MRI for ABE. Looking forward, we aim to integrate more advanced MRI technologies and larger datasets in our future research. This will enable us to further validate the effectiveness and reliability of our methodology, continuing to contribute to the field of medical imaging and diagnostics.

2. Materials and Methods

2.1. Study Subjects

The data for this study were obtained from routine clinical examinations conducted at Wuxi People’s Hospital between 2020 and 2022, with all research protocols receiving approval from the Clinical Research Ethics Committee. We recruited 285 newborns with high bilirubin levels, among whom 145 were diagnosed with ABE, and 140 were identified as non-ABE cases. Experienced pediatricians confirmed the diagnosis for all participants based on clinical examination results and the Neonatal Behavioral Neurological Assessment (NBNA). The NBNA, developed by Professor Bao Xiulan from Peking Union Medical College Hospital, combines the strengths of the American Brazelton Neonatal Behavioral Assessment Scale [25] and the French Amiel-Tison neonatal neuromotor assessment method [26]. This assessment is primarily used to evaluate the integrity of neonatal nervous system development, covering five domains: behavioral capability, passive muscle tone, active muscle tone, primitive reflexes, and overall assessment. The assessment comprises 20 items, with a total score of 40 points. Newborns diagnosed as non-ABE did not exhibit the corresponding clinical neurological symptoms.

2.2. MRI Acquisition

MRI images were acquired using a 1.5 T Siemens MRI scanner (Siemens Shenzhen Magnetic Resonance Ltd., Shenzhen, China), adhering to experimental requirements. T1-weighted images were obtained with the following parameters: TR/TE of 200/4.8 ms, a slice thickness of 4 mm for 20 slices, a flip angle of 90°, a matrix size of 272 × 288, and a field of view (FOV) of 217 × 230 mm. For T2-weighted images, the parameters were TR/TE of 2800/98 ms, a slice thickness of 4 mm for 20 slices, a flip angle of 150°, a matrix size of 256 × 256, and an FOV of 230 × 230 mm. All images were manually inspected by pediatricians to ensure they met quality standards for subsequent analysis.

2.3. Image Preprocessing

During the image preprocessing, MRI images underwent several processing steps. First, the FSL v7.0 SynthStrip module was used for skull stripping of both T1- and T2-weighted images [27]. To standardize inter-image variability and enhance model robustness and computational efficiency, images were normalized to a grayscale range of 0–1 and resized to 128 × 128 pixels. From each T1 and T2 image modality, 6 consecutive slices surrounding the GP were selected as the 3D input for the model, resulting in a 3D tensor of intensity values with a size of 128 × 128 × 6. All preprocessing tasks were performed using Python 3.8 and FSL 6.0.6 software on an Ubuntu 20.04 system.

To address the challenges of limited training data and potential overfitting, we incorporated robust data augmentation techniques into our methodology. These involved random horizontal and vertical translations of images ranging from −10 to 10 pixels, rotations between −30 and +30 degrees, and scaling of images from 0.8 to 1.2 times their original size (Figure 1).

2.4. Deep Learning Framework

In this study, we developed a multi-modal image classification network based on the 3D ResNet18 architecture (Figure 2). The primary function of 3D ResNet18 in our framework is the extraction of features from T1- and T2-weighted MRI images. We chose 3D ResNet18 for its inherent capability to handle three-dimensional data, crucial for accurately interpreting the complex structures within MRI images. Unlike other models primarily designed for 2D image processing, 3D ResNet18 employs 3D convolutional kernels, effectively capturing spatial details across all dimensions. This feature makes it particularly suitable for medical imaging applications where precision is of the utmost importance.

To enhance the model’s feature extraction capabilities, we strategically incorporated spatial and channel attention modules within different residual blocks of the 3D ResNet18 network [28]. Concurrently, CentralNet, proven to be effective in integrating features from different modalities, as demonstrated by Valentin Vielzeuf et al. [24], is deployed between the two feature extraction pathways to integrate stage-specific features from T1 and T2 modalities. This architecture offers a more dynamic and effective feature integration compared to traditional feature fusion methods. The multi-layered approach of CentralNet allows for a richer and more nuanced representation of data, making it an optimal choice for the complex requirements of medical image analysis.

This design leads to a robust model, fine-tuned through joint multi-task training, and specifically aimed at the classification of ABE. This method represents a harmonious integration of advanced attention mechanisms with multi-modal feature fusion, significantly boosting the model’s diagnostic accuracy.

In our methodology, attention modules are integrated within the residual blocks of the 3D Res-Net18 network (Figure 3 and Figure 4) [28]. These modules enhance the model’s focus across spatial and channel dimensions, substantially enhancing its feature extraction capacity. The spatial attention module uses maximum and average pooling operations to generate spatial weight maps, directing the model’s attention towards crucial image regions for classification. The channel attention module computes maximum and average pooling along the channel axis, producing two sets of shared weight coefficients for the channels. These coefficients are processed through a shared-weight multi-layer perceptron (MLP) and subsequently aggregated via weighted summation to derive unique channel weight coefficients. This dual attention mechanism ensures balanced and prioritized comprehension of spatial and channel-specific features, essential for high precision in classification tasks.

CentralNet’s architecture integrates features from different modalities by taking the weighted sum of layers from the corresponding unimodal networks and its own preceding layers as input. Each modality is processed by a distinct deep convolutional network, with a central network integrating these modality-specific features. Traditional methods of multi-modal feature fusion involve independently extracting features from each modality and then combining them. In contrast, the CentralNet structure enables mutual supervision and learning from each other during the feature extraction process across different modalities. This method represents progress in addressing the complexity of multi-modal data fusion, offering a flexible and effective framework for integrating various data types.

2.5. Model Training and Evaluation

In this study, the model’s generalization ability was evaluated by dividing the dataset into training, validation, and test sets in a 6:2:2 ratio. The dataset underwent random partitioning to ensure unbiased evaluation. Subsequently, the evaluation metrics were calculated based on the average results from 50 tests, providing robust insights into the model’s performance under diverse conditions.

These evaluation metrics, including classification accuracy, Area Under the Curve (AUC), sensitivity, specificity, recall rate, and F1 score, were utilized to comprehensively assess the model’s performance. Each metric offers unique insights into different aspects of the model’s predictive capabilities. For instance, accuracy measures overall correctness in predictions, while precision and recall focus on correctly identifying positive instances. Specificity evaluates the model’s ability to identify negative instances accurately, and the F1 score provides a balanced measure of precision and recall. The calculations of these metrics were based on Equations (1)–(5), where TP, FP, TN, and FN represent true positive, false positive, true negative, and false negative cases, respectively. The performance indicators are represented as the mean value plus or minus the standard deviation from these tests.

$$\mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{FP}+\mathrm{TN}+\mathrm{FN}}$$

(1)

$$\mathrm{Sensitivity}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(2)

$$\mathrm{Specificity}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}$$

(3)

$$\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

(4)

$$\mathrm{F}1 \mathrm{Score} =2\times \frac{\mathrm{Precision}\times \mathrm{Sensitivity}}{\mathrm{Precision}+\mathrm{Sensitivity}}$$

(5)

In our study, we conducted a two-part experimental investigation to assess the impact of attention modules and CentralNet on MRI data classification. The first part focused on single modality experiments, where the model was individually fed with either T1 or T2 modality images. The primary objective here was to compare the classification performance of models without the attention module against those equipped with it. This comparison aimed to elucidate the specific contribution of an AM in enhancing the model’s ability to classify MRI images based on single modalities.

Following the single modality experiments, we delved into multi-modal classification. Initially, we adopted the early fusion method as described in Wu et al.’s study [22], where T1 and T2 MRI modalities were concatenated at the outset before being input into the model. This approach set the baseline for our multi-modal analysis. Subsequently, we implemented a later fusion strategy, differing from early fusion by merging features from T1 and T2 modalities at a later stage in the model’s processing. This step was crucial in understanding the benefits of fusing features at different stages of the model. The experiments were further extended by integrating attention modules into the later fusion framework. Finally, our proposed model, incorporating both an AM and CentralNet, was evaluated to determine its efficacy in optimizing classification performance in a multi-modal MRI context.

To achieve effective model parameter initialization and improved training efficiency, we utilized transfer learning techniques. The pre-trained weights from ImageNet were downloaded from the PyTorch website (https://download.pytorch.org/models/resnet18-5c106cde.pth, accessed on 7 December 2023) to initialize the weights of the feature extraction module in our model. The hyperparameters for training were set as follows: an initial learning rate of 0.0001, maximum epoch of 200, and a minibatch size of 32. The Adam algorithm was used to train the model. This experiment was conducted using Python 3.10 on a Windows 11 operating system.

3. Results

The demographic and clinical characteristics of the subjects are summarized in

Table 1. The demographic and clinical characteristics of the subjects.

Clinical Features	ABE (n = 145)	Non-ABE (n = 140)	p Value
Gender (male/female)	79/66	85/55	0.287 1
Weight (kg)	3.78 ± 0.68	3.98 ± 1.06	0.138 2
Age (days)	9.89 ± 5.67	15.76 ± 11.69	<0.001 2

¹ The p value is the two-sided asymptotic significance from the Pearson’s Chi-Squared test. ² For weight and age, which did not pass normality tests (p < 0.05 in both Kolmogorov–Smirnov and Shapiro–Wilk tests), the p values are from the non-parametric Mann–Whitney U test.

, which compares the group of 145 subjects with ABE to the group of 140 subjects with non-ABE. When considering the gender distribution, the ABE group had 79 males and 66 females, while the non-ABE group consisted of 85 males and 55 females. However, the difference in the gender ratio between the groups was not statistically significant, as indicated by a p value of 0.287 (Pearson’s Chi-Squared test). The average weight of the subjects in the ABE group was 3.78 ± 0.68 kg, slightly lower than the average weight of 3.98 ± 1.06 kg in the non-ABE group, but this difference was not statistically significant (p = 0.138, Mann–Whitney U test). A significant difference was observed in the age of the subjects, with the ABE group exhibiting a younger age profile (9.89 ± 5.67 days) compared to the non-ABE group (15.76 ± 11.69 days). The age difference was statistically significant (p < 0.001, Mann–Whitney U test).

In the assessment of the single-modality MRI data, the incorporation of an attention module (AM) resulted in a significant improvement in all the performance metrics for both the T1- and T2-weighted images (see

Table 2. Performance evaluation of single-modal MRI data with and without attention module across various metrics.

Module	MRI Modality	Accuracy	AUC	Sensitivity	Specificity	Precision Score	F1 Score
None	T1	0.606 ± 0.041	0.636 ± 0.047	0.655 ± 0.114	0.555 ± 0.123	0.609 ± 0.046	0.624 ± 0.057
None	T2	0.688 ± 0.034	0.736 ± 0.04	0.779 ± 0.087	0.593 ± 0.065	0.666 ± 0.03	0.716 ± 0.042
AM	T1	0.639 ± 0.04	0.682 ± 0.037	0.688 ± 0.09	0.588 ± 0.068	0.634 ± 0.034	0.657 ± 0.052
AM	T2	0.738 ± 0.039	0.796 ± 0.025	0.764 ± 0.083	0.711 ± 0.048	0.733 ± 0.029	0.746 ± 0.048

AM = attention module; None = model without attention module.

). Specifically, the application of the attention module on the T1-weighted MRI resulted in an accuracy of 0.639 ± 0.04, an AUC of 0.682 ± 0.037, and a sensitivity of 0.688 ± 0.09. For the T2-weighted images, the performance was even more significant, with an accuracy of 0.738 ± 0.039 and an AUC reaching 0.796 ± 0.025. This improvement suggests that the attention module offers a significant advantage in interpreting MRI data.

In our multi-modal experiment, we rigorously evaluated five distinct models for classifying combined T1 and T2 MRI modalities (

Table 3. Impact of attention modules and CentralNet on classification metrics using combined T1 and T2 MRI modalities.

Module	MRI Modality	Accuracy	AUC	Sensitivity	Specificity	Precision Score	F1 Score
None (Early Fusion)	T1 + T2	0.648 ± 0.043	0.693 ± 0.039	0.709 ± 0.078	0.586 ± 0.048	0.639 ± 0.033	0.671 ± 0.048
AM (Early Fusion)	T1 + T2	0.663 ± 0.046	0.738 ± 0.034	0.714 ± 0.103	0.611 ± 0.062	0.655 ± 0.034	0.68 ± 0.056
None (Later Fusion)	T1 + T2	0.728 ± 0.054	0.77 ± 0.024	0.747 ± 0.103	0.708 ± 0.035	0.724 ± 0.038	0.733 ± 0.067
AM (Later Fusion)	T1 + T2	0.766 ± 0.013	0.804 ± 0.027	0.795 ± 0.055	0.735 ± 0.059	0.759 ± 0.027	0.775 ± 0.017
AM + CentralNet	T1 + T2	0.845 ± 0.018	0.913 ± 0.02	0.954 ± 0.069	0.732 ± 0.077	0.792 ± 0.048	0.862 ± 0.017

AM = attention module; None = model without attention module; fusion strategy: early fusion, later fusion, CentralNet.

). These models encompassed various fusion strategies and attention mechanisms to optimize the classification accuracy. We first examined the “None (Early Fusion)” model, following Wu et al.’s early fusion strategy [22], which concatenates T1 and T2 modalities before inputting into the model. This baseline approach achieved an accuracy of 0.648 ± 0.043, a sensitivity of 0.709 ± 0.078, and a specificity of 0.586 ± 0.048. The subsequent integration of attention modules (AMs) into the early fusion framework, denoted as “AM (Early Fusion)”, yielded noticeable improvements across all the metrics, with an accuracy of 0.663 ± 0.046, a sensitivity of 0.714 ± 0.103, and a specificity of 0.611 ± 0.062.

Following early fusion, we explored the “None (Later Fusion)” model, wherein the T1 and T2 features were combined at a later stage in the model’s processing. This later fusion strategy outperformed early fusion, achieving an accuracy of 0.728 ± 0.054, a sensitivity of 0.747 ± 0.103, and a specificity of 0.708 ± 0.035. Integrating attention modules (AMs) into the later fusion framework further improved the performance, resulting in the “AM (Later Fusion)” model, which attained an accuracy of 0.766 ± 0.013 and a sensitivity of 0.795 ± 0.055. The most significant advancements, however, were observed with the “AM + CentralNet” model, where attention modules were combined with CentralNet for feature fusion. This approach achieved the highest performance, with an accuracy of 0.845 ± 0.018 and sensitivity of 0.954 ± 0.069, underscoring the effectiveness of attention mechanisms and advanced fusion techniques in enhancing multi-modal MRI classification.

Figure 5 presents the performance metrics of different models for ABE classification. Panel (a) shows the confusion matrix for our proposed model, illustrating the number of true positives (ABE correctly identified as ABE) at 26 and true negatives (non-ABE correctly identified as non-ABE) at 23, indicating a strong predictive capability. However, there are instances of misclassification, with 7 false negatives (ABE incorrectly identified as non-ABE) and 1 false positive (non-ABE incorrectly identified as ABE), which suggests areas for potential improvement in the model’s sensitivity.

In our study, we meticulously evaluated the processing times across different computational models during various stages of our experiment, including preprocessing, training, and testing (

Table 4. Comparative processing times in different computational models.

Procedure	None (Early Fusion)	None (Later Fusion)	AM (Later Fusion)	Ours Model
Preprocessing	59.0 (s/sample)	59.0 (s/sample)	59.0 (s/sample)	59.0 (s/sample)
Training	2.10 (s/epoch)	2.20 (s/epoch)	3.80 (s/epoch)	4.50 (s/epoch)
Testing	0.25 (s/epoch)	0.27 (s/epoch)	0.35 (s/epoch)	0.60 (s/epoch)

). The preprocessing stage displayed a uniform duration across all the models, with each taking approximately 59 s per sample. During the training and testing phases, we noted significant differences in the processing times depending on whether an attention module was used and the type of fusion strategy employed. The ‘None’ model, which did not include an attention module, exhibited the shortest processing times. Conversely, models using either early fusion or later fusion strategies, especially those incorporating an attention module, required longer processing times due to the increased computational complexity.

The presence of an AM and the choice of fusion strategy, particularly later fusion, led to a modest increase in the processing time but significantly enhanced the model’s analytical capabilities. This trade-off is particularly evident in the accuracy and reliability improvements in multi-modal MRI analysis for diagnosing Acute Bilirubin Encephalopathy (ABE). Our findings highlight the effectiveness of these approaches, balancing computational efficiency with advanced performance capabilities.

4. Discussion

Bilirubin-induced brain injury is distinctive, with the pallidum, basal ganglia, substantia nigra, hippocampus, thalamic nuclei, and putamen being notably and selectively affected in a symmetrical pattern [9,13,29]. The most common and characteristic finding in ABE is bilateral symmetric high signals on T1-weighted MRI. In cases of severe neurological damage caused by bilirubin, MRI can, to some extent, detect signal changes caused by the injury. This change is a long-term progression, characterized by bilateral symmetric high signals in T1-weighted sequences transitioning to high signals in T2-weighted sequences and fluid-attenuated inversion recovery (FLAIR) sequences in specific areas of the brain [10,13,30]. However, not all patients with hyperbilirubinemia exhibit typical radiological findings that are visible to the naked eye in conventional MRI sequences commonly used in current clinical practice. This is particularly true for patients who are not critically ill or are in the early stages of the condition. Identifying ABE or hyperbilirubinemia using MRI based on traditional radiological experience is quite challenging in clinical practice.

In this study, we proposed a deep learning framework based on multi-modal feature fusion and attention modules to distinguish ABE from non-ABE conditions in newborns. The application of the attention module and CentralNet has demonstrated the potential for advanced diagnostic accuracy in the evaluation of MRI data. The attention module’s ability to refine the focus on salient features within the imaging data is evidenced by the observed improvements in sensitivity and precision scores. Furthermore, the incorporation of CentralNet has notably enhanced the model’s accuracy and AUC. This is largely attributed to CentralNet’s ability to effectively uncover inter-modal associative information. During the feature extraction phase, there is mutual supervision and learning across modalities, thereby more efficiently distilling critical diagnostic information from the complex imaging data. This collaborative process facilitated by CentralNet leads to a more insightful synthesis of diagnostic features, improving the model’s overall diagnostic capability.

Although the specificity did not improve at the same rate, it remained relatively consistent, which implies that the models did not lose their ability to identify true negatives correctly. This is particularly important in clinical practice to avoid false positives that could lead to unnecessary interventions. The notable increase in the F1 score with the use of both the attention module and CentralNet indicates a balanced improvement in the precision and recall, which is critical for the reliable classification of medical imaging data.

However, our study still has certain limitations. First, the source of our patient sample is singular and the sample size is relatively small. Although we have employed different strategies, such as data augmentation and transfer learning, to address overfitting, these measures may not completely offset the effects of the small sample size, which could affect the model’s ability to generalize. This limitation may also impede the attainment of a consistent sample distribution across training, validation, and test datasets. Secondly, as illustrated in Table 1, there is a notable age difference between subjects in the ABE group and those in the non-ABE group. Given the rapid and distinctive development of the infant brain, we cannot entirely dismiss the impact of different stages of brain development on our findings. Furthermore, the incorporation of additional modalities, such as Magnetic Resonance Spectroscopy (MRS), perfusion MRI, and comprehensive clinical data, could potentially improve the diagnostic effectiveness of our approach [31,32]. Although the classification accuracy of our current model is not yet suitable for clinical deployment, this study lays a valuable foundation for future investigations into the use of multi-modal MRI for diagnosing ABE.

In future studies, we aim to stay at the forefront of deep learning and medical image analysis by focusing on innovative models, especially Vision Transformers (ViTs) [33]. ViTs, with their advanced performance in image data processing, use a novel method of image segmentation into patches and a self-attention mechanism. This approach is critical in medical imaging, enabling the detection of minute but diagnostically crucial variations. ViTs excel in modeling long-range dependencies within images, offering a substantial advantage over traditional Convolutional Neural Networks (CNNs), especially for complex medical image analysis. Their strong transfer learning capabilities are well suited for specific medical imaging tasks, potentially increasing diagnostic efficiency [34]. Furthermore, the interpretability aspect of ViTs, through their attention mechanism, is invaluable in clinical settings for understanding model decisions. Integrating Vision Transformers into our research, we aim to create more accurate, reliable, and interpretable diagnostic models for ABE diagnosis.

5. Conclusions

In our study, we developed a multi-modal MRI classification network based on the 3D ResNet18 architecture. This network exhibited outstanding capability in extracting features from T1- and T2-weighted MRI images. Integrating spatial and channel attention modules within various residual blocks, our network effectively concentrated on key image features. CentralNet, a pivotal component of our architecture, facilitated mutual supervision and learning in the cross-modal feature extraction process, enhancing the model’s precision in classifying ABE. The rationality of our model structure and the efficacy of combining attention mechanisms with CentralNet were validated through several control experiments, demonstrating the significance of this approach in medical imaging data analysis. Future research could investigate the integration of more advanced MRI technologies and larger datasets to further confirm the effectiveness and reliability of our methodology.