Bridging Modalities: A Multimodal Machine Learning Approach for Parkinson’s Disease Diagnosis Using EEG and MRI Data

Abstract

Parkinson’s disease (PD) is a slowly progressing neurological disorder with symptoms that overlap with those of other conditions, making early detection and accurate diagnosis vital for effective treatment and a patient’s quality of life. Symptoms such as tremors, stiffness, slow movements, and balance issues, along with psychiatric manifestations, are typical of PD. This study introduces a groundbreaking approach to PD diagnosis, utilizing a multimodal machine learning framework that integrates Electroencephalography (EEG) and Magnetic Resonance Imaging (MRI) data. Focusing on the early detection and accurate classification of PD, the proposed research leverages the distinct yet complementary nature of EEG and MRI datasets to enhance diagnostic precision. We employed a robust algorithmic strategy, including LightGBM and machine learning techniques, to analyze the complex patterns inherent in neurological data. The key steps of the proposed research are preprocessing and feature extraction from both EEG and MRI modalities, followed by their fusion using Principal Component Analysis (PCA) for dimensionality reduction. The fused dataset was then analyzed using a LightGBM model and validated through a 10-fold cross-validation process to ensure reliability and stability. The model’s efficacy was further tested on independent datasets, demonstrating its robustness across diverse patient demographics. The obtained results showcased an accuracy of 97.17%, sensitivity of 96.58%, and specificity of 96.82% in PD classification, outperforming traditional multimodal as well as single-modality diagnostic methods. The integration of EEG and MRI data provided a more comprehensive view of the neurophysiological and neuroanatomical changes associated with PD. Additionally, the use of advanced machine learning algorithms allowed for a nuanced analysis, capturing subtle patterns indicative of early PD stages.

1. Introduction

Parkinson’s disease (PD) is a relentless neurological condition marked by the gradual depletion of dopamine in the brain, which is a chemical crucial for messaging within the basal ganglia—the brain region that governs movement and coordination [1]. This depletion occurs as the dopamine-producing cells in the basal ganglia die off or malfunction [2]. Symptoms of PD range from tremors, movement restrictions or slowness (bradykinesia), compromised balance and posture, involuntary movements (dyskinesia), and muscle stiffness to altered speech and writing abilities [3].

Symptoms of PD can vary widely among individuals, including the speed at which the disease progresses, as depicted in Figure 1. The most common “hallmark” symptoms of PD are the following:

• Bradykinesia: This refers to a general slowness in physical movements [4]. Affected individuals may experience impaired dexterity, reduced frequency of blinking, drooling, and a lack of facial expressions.
• Tremor at Rest: This symptom involves involuntary shaking, typically noticeable when the body is at rest, which diminishes during purposeful movement [5]. It often starts on one side of the body, with the hand being a common initial site.
• Rigidity: This is characterized by an involuntary increase in muscle tone, leading to stiffness in the muscles [6].
• Postural Instability: Individuals with PD often have a compromised sense of balance, which can lead to frequent falls [7]. To counter this instability, many patients adopt a stooped posture by lowering their center of gravity.

The World Health Organization indicates that approximately 10 million people worldwide suffer from PD [8]. Early diagnosis is crucial, yet many cases remain undetected until the disease progresses to an advanced, incurable stage [9]. In 2015, about 6.2 million individuals were affected by PD, leading to 117,400 deaths [10]. A significant challenge in combating PD is the high cost and limited accuracy of current diagnostic tests [11,12]. This situation underscores the critical need for an affordable, effective, and precise diagnostic tool for early-stage PD. Such a development could enable timely treatment, offering a chance to cure the disease before it progresses beyond treatment.

Diagnosing PD is challenging due to the lack of definitive clinical tests like blood work, often leading to late detection, particularly in individuals younger than 60 [13]. Moreover, other health conditions and medications can also mimic Parkinson’s disease symptoms, so accurate diagnosis by a physician is crucial [14,15,16]. The variability of symptoms among individuals means that no single test can conclusively diagnose Parkinson’s disease (PD). During a physical exam, potential PD patients may be asked to perform specific movements to assist in diagnosing. For instance, when individuals with Parkinson’s are instructed to touch their noses with their hands, their tremor often reduces or vanishes. Additionally, they typically struggle with tasks involving rapid, alternating movements, like repeatedly placing their hands on their thighs and flipping them over quickly.

As an alternative to this, machine learning methods are increasingly being utilized in the diagnosis of PD, offering significant advancements in early and accurate detection [17]. These techniques involve analyzing large datasets of medical information, including patient symptoms, genetic data, and neuroimaging results. One common approach is the use of machine learning algorithms, which are trained on datasets of known PD cases and controls [18]. These models learn to identify patterns and correlations in the data that are indicative of PD. For instance, machine learning can be used to analyze speech patterns or motor skills, which are often affected in PD patients. By processing complex data and identifying subtle anomalies that might be overlooked in standard diagnostic procedures, machine learning provides a powerful tool for clinicians, enhancing the accuracy of PD diagnosis and enabling earlier intervention, which can be crucial in managing the disease’s progression.

Lately, deep learning approaches have significantly outperformed traditional algorithms in classification tasks, with convolutional neural networks (CNNs) setting new benchmarks for accuracy [19]. These algorithms have become widely adopted for sorting images, audio files, and videos by identifying and leveraging unique data patterns for classification. The widespread accessibility and user-friendliness of CNNs make them a top pick for such tasks. They are often more effective, partly due to their ability to utilize transfer learning. This technique adapts pre-trained networks to new problems by adding new layers, allowing for broad applications across various scenarios, including ResNets, EfficientNets, and MobileNets, among others.

Most of the PD diagnosis methods rely on either EEG or MRI, but all such methods require specific diagnostic tools, such as EEG or MRI, which have a specific scope and provide limited types of information [20]. For instance, while EEG excels in capturing the brain’s electrical activity, it lacks the spatial resolution to detail brain structures, a gap that MRI can fill. However, MRI does not track real-time neuronal activity. Furthermore, EEG, which tracks and records brain wave patterns, is highly sensitive to the brain’s electrical activity, providing real-time data on neuronal functioning. However, it lacks spatial resolution. MRI, in contrast, yields high-resolution images of the brain’s structure, including detailed views of brain tissue, but does not capture the moment-to-moment activity that EEG can. By integrating the functional insights of EEG with the structural details provided by MRI, this hybrid method can detect subtle neurological changes early on and monitor disease progression more effectively. This combined approach compensates for the limitations inherent in each technique individually, leading to a more nuanced and accurate understanding of complex brain disorders.

Research Contribution

The key contributions of the proposed research are as follows:

• By combining the real-time, functional insights provided by EEG with the detailed structural information from MRI, a hybrid model can offer a more accurate diagnosis. This integrative approach leverages the strengths of both methods, potentially leading to earlier and more reliable identification of PD, especially in cases where symptoms are ambiguous or in their early stages.
• The proposed multimodal technique allows a more holistic view of the brain’s functioning and structure. This comprehensive understanding is crucial in tracking the progression of PD, enabling clinicians to observe changes over time in both brain activity and physical structure. This can be instrumental in tailoring treatment plans and monitoring the effectiveness of therapeutic interventions.
• PD manifests differently in each individual. A hybrid diagnostic approach can provide a more personalized assessment of the disease’s impact on a specific patient. This individualized understanding is key to developing personalized treatment strategies, ensuring that each patient receives care that is tailored to their unique presentation of the disease.
• Through rigorous experimental evaluations and comparative analyses with established benchmark methods, the proposed method has demonstrated superior performance, achieving an impressive accuracy rate of 97% for the PD-DS-I and PD-DS-II datasets.

The structure of this paper is as follows: Section 2 reviews past research on Parkinson’s detection. Section 3 outlines the materials and methods used in our study. Section 4 presents the experimental outcomes and provides a comparative evaluation of our model against pre-existing ones. Section 5 concludes by summarizing our work and findings.

2. Literature Review

Several researchers, as detailed in Table 1, have implemented machine learning and deep learning techniques to diagnose Parkinson’s disease (PD). These methods include the analysis of vocal features and brain imaging, along with certain specific drawings like meanders, spirals, and waves. Deep learning has gained prominence for its remarkable precision in detecting early stages of PD, particularly in medical imaging applications. For instance, Chintalapudi et al. [21] used the Synthetic Minority Over-sampling Technique (SMOTE) on 195 voice samples to artificially expand the size of their dataset. SMOTE involves creating additional data points to balance the dataset by amplifying the representation of the minority class. The goal of this oversampling was to form a new dataset resembling the original in-class distribution but with a greater percentage of minority-class samples. Following this, LSTM networks were utilized to enhance the categorization of the disease into specific types. Kemal et al. [22] also applied this oversampling strategy to classify Parkinson’s disease using vocal signals. It is critical to recognize, though, that such sampling techniques can introduce noise into the dataset if the chosen samples do not accurately reflect the true distribution. In their study, 50% of the data were allocated for training and testing, but this method resulted in a relatively modest accuracy rate of 94.8%.

Quan et al. [23] conducted research employing a bi-directional LSTM deep learning model with two LSTM layers, consisting of 20 and 200 units, respectively. The study achieved a 75% accuracy and an 80% F1 score in their experiments. In a separate study, Oh et al.’s [24] research involved a dataset from 20 patients and applied a CNN. Despite making 361 prediction errors, their model attained an accuracy of 88%. Wodzinski et al. [25] also focused on using voice signals to predict PD, employing an LSTM model. They collected their dataset from a hundred patients, split evenly between healthy individuals and those with PD. After processing the dataset and applying their deep learning model, they achieved a notable accuracy of 91%.

Fang et al. [26] proposed an enhanced version of the K-Nearest Neighbors (KNN) algorithm, incorporating entropy weight for the detection of PD. The results showed that the improved KNN algorithm, with its entropy weight consideration, demonstrated a significant increase in accuracy when compared to traditional methods. This finding highlighted the potential of the modified KNN approach in improving the accuracy of PD detection. Kuplan et al. [27] introduced a novel method for classifying symptoms of PD using MRI scans. The primary objective of their study was to delve deeper into clinical data to enhance the effectiveness of artificial intelligence in detecting PD. The integration of these techniques resulted in a model that demonstrated exceptional performance across all classification tasks. This advancement underscores the potential of AI-driven methods in improving the diagnosis and understanding of PD.

Tuncer and Dogan [28] developed a unique multi-pooling technique for classification, known as the octopus-based method, which utilized eight different pooling methods. This approach was applied to solve three classification problems: Gender, PD, and a combined Gender and PD classification. For feature extraction and selection, the authors employed techniques such as Singular Value Decomposition (SVD) and Neighborhood Component Analysis (NCA). In comparison, KNN was found to be particularly effective in solving the PD classification problem with high accuracy. Remarkably, the authors were able to address all three classification challenges using only 32 features, demonstrating the efficiency and effectiveness of their novel approach.

Table 1. Comparative Analysis of Core Methodologies.

Reference	Core Methodology	Dataset Used	Accuracy
Gazda et al. [29]	CNN with LSTM	PaHaW and NewHandPD	89.23% on PaHaw and 91.11% on NewHand PD
Mohaghegh and Gascon [30]	Vision Transformer with ImageNet	Hand Written Dataset	92.11%
Fratello et al. [31]	Mann_Whitney test, SVM and KNN	31 patient data of Casa di Cura Le Terrazze institute	71.1% for SVM and 75.62% for KNN
Sunil et al. [32]	Chi Test, Extra Tree Classifier, RNN, and Boosting	8 patients with 233 instances and 23 features	ROC Accuracy 98.7%
Sayed et al. [33]	XGBoost, Decision Tree, SVM	UCI dataset with 195 instances and 24 features	ROC Accuracy 96.2% and Sensitivity 100
Nisashi et al. [34]	Deep learning and Neuro-fuzzy, PCA, and Deep Belief Network	Real World Dataset 5875 record with 16 features	ROC Accuracy 88.7%

Table 1. Comparative Analysis of Core Methodologies.

Reference	Core Methodology	Dataset Used	Accuracy
Gazda et al. [29]	CNN with LSTM	PaHaW and NewHandPD	89.23% on PaHaw and 91.11% on NewHand PD
Mohaghegh and Gascon [30]	Vision Transformer with ImageNet	Hand Written Dataset	92.11%
Fratello et al. [31]	Mann_Whitney test, SVM and KNN	31 patient data of Casa di Cura Le Terrazze institute	71.1% for SVM and 75.62% for KNN
Sunil et al. [32]	Chi Test, Extra Tree Classifier, RNN, and Boosting	8 patients with 233 instances and 23 features	ROC Accuracy 98.7%
Sayed et al. [33]	XGBoost, Decision Tree, SVM	UCI dataset with 195 instances and 24 features	ROC Accuracy 96.2% and Sensitivity 100
Nisashi et al. [34]	Deep learning and Neuro-fuzzy, PCA, and Deep Belief Network	Real World Dataset 5875 record with 16 features	ROC Accuracy 88.7%

From the above literature, it has been concluded that PD is a debilitating neurological disorder that greatly impacts the quality of life of those affected. Accurate and early diagnosis is paramount for effective management and treatment of the disease. The current body of research, as referenced in various studies, employs a variety of methods for collecting data from both healthy individuals and those with PD. While these approaches have shown promise in enhancing the diagnosis of PD, there is a consensus in the scientific community that further research is needed to identify the algorithms that provide the highest levels of precision and accuracy in diagnosing PD.

The focus of much of the existing research is on optimizing the performance of various leading deep learning models, which are at the forefront of advancements in artificial intelligence and medical diagnostics. Simultaneously, there is continued relevance and use of traditional machine learning algorithms in this research area. However, the statement suggests that there is a significant opportunity for research in the domain of adaptive heuristic algorithms, like genetic algorithms, which are known for their ability to evolve solutions to complex problems. Investigating the potential of these heuristic algorithms in the context of PD detection could open new avenues for analyzing complex data patterns and enhance the diagnostic process, ultimately leading to more precise and accurate detection of PD.

3. Proposed Methodology

Creating a multimodal method for diagnosing PD by analyzing EEG and MRI data entails a comprehensive strategy that utilizes various machine learning techniques. The combination of EEG feature extraction using Flexible Analytic Wavelet Transform and MRI feature extraction utilizing Functional MRI (fMRI) with the General Linear Model and classification through algorithms like LightGBM is pivotal in advancing medical diagnostics and cognitive neuroscience. By extracting intricate patterns from EEG and fMRI data, it enables early detection and personalized treatment. This synergy offers a comprehensive insight into brain function and connectivity, fostering innovation in neuroimaging and signal-processing techniques. Ultimately, this integrated approach holds promise for improving patient outcomes and deepening our understanding of the complexities of the human brain. The proposed work, as depicted in Figure 2, is an attempt to diagnose PD from multimodal data. The proposed model begins with data collection, which includes MRI and EEG images. Next, data preprocessing removes distortion from these images. Following that, EEG features are extracted using wavelet transformation, and MRI features are extracted using functional MRI and the General Linear Model. Both extracted features are fused using PCA. Finally, LightGBM is employed for classification.

3.1. Data Collection

The process of data collection for PD diagnosis using EEG and MRI involves a meticulous and multi-faceted approach. EEG (Electroencephalography) data collection entails recording the electrical activity of the brain using sensors placed on the scalp. This method captures the brain’s electrical fluctuations, which are crucial in understanding the neural dynamics associated with PD.

MRI (Magnetic Resonance Imaging) provides images of the brain’s structure. In PD diagnosis, MRI is particularly useful for visualizing changes in brain regions affected by the disease, such as the basal ganglia. The combination of EEG and MRI data offers a comprehensive view of both the functional and structural aspects of the brain when gathering these data. The combined data collection approach in this research enhances the understanding of PD’s impact on the brain, leading to a more accurate diagnosis and tailored treatment strategies. Figure 3 shows the snippets of EEG, and Figure 4 depicts MRI of PD and non-PD patients. In a normal EEG, a balanced distribution of various frequency bands, including delta, theta, alpha, beta, and gamma, is observed depending on the individual’s state of consciousness. However, in Parkinson’s disease, EEG recordings often show an increase in beta frequency band activity, particularly in the range of 13–30 Hz, especially during resting conditions. This exaggerated beta band activity, known as “resting tremor,” is a hallmark feature of Parkinson’s disease and distinguishes it from normal brain activity patterns. A clear MRI image typically displays a healthy brain with normal anatomical structures and no signs of pathology. In contrast, an MRI Image of a brain affected by Parkinson’s disease may reveal specific abnormalities. These can include changes in the size and shape of certain brain regions, such as the substantia nigra, as well as the presence of characteristic abnormalities.

3.2. Data Preprocessing

Two different streams of preprocessing have been applied due to the different nature of the data. In the very first approach, the preprocessing of EEG has been performed, in which the EEG data are prone to various types of noise and artifacts, including eye blinks, muscle movements, and external electromagnetic interferences, as shown in Figure 5.

However, EEG preprocessing does not have a universally agreed-upon pipeline, as it remains an area of ongoing research [35]. The optimal preprocessing steps can vary significantly depending on the specific characteristics of each EEG signal. It is important to carefully consider the properties of the dataset in question to determine which preprocessing methods will most effectively enhance the validity and interpretability of the data. Each dataset may require a unique combination of steps tailored to its specific needs and the objectives of the study. The working of EEG Preprocessing is shown in Algorithm 1.

Algorithm 1: EEG Preprocessing

Input: X← EEG data matrix Output: Xpro. ←EEG data 1. Artifact Removal using ICA       A, S ← ICA(X)    // Decompose X       S_clean ← S—Artifacts // Remove artifact components from S       X_clean ← A−1 * S_clean // Reconstruct clean EEG data 2. Remove Bad Channels       C_bad ← IdentifyBadChannels(X_clean) // Identify bad channels       X_clean_good ← X_clean—C_bad    // Exclude bad channels 3. Apply Filters       X_filtered ← Filter(X_clean_good)   // Apply necessary filters (e.g., low-pass, high-pass) 4. Output Processed Data       Xpro. ← X_filtered        // Final processed EEG data

In this research, ICA (Independent Component Analysis) [36] has been utilized to identify and extract eye blinks and movements, subsequently removing these elements from the EEG signals. Another critical aspect of preprocessing involves identifying “bad” channels in the EEG data, which do not provide reliable information. It is crucial to exclude these channels early in the analysis process, as retaining them can adversely affect further analyses. For this purpose, visualization methods have been employed. Additionally, various types of filters are applied to the signals to refine the data further. These include a low-pass filter [37] that retains frequencies below a certain threshold (low frequencies “pass through”), while higher frequencies above this threshold are filtered out. A high-pass filter [37] that operates inversely to the low-pass filter allows only high frequencies to remain, eliminating those below a specified value. Whereas the notch filter [38] has been designed to remove a specific single frequency (noise), multiple notch filters can be used together to eliminate a particular set of single frequencies. The working of MRI Preprocessing is shown in Algorithm 2.

Algorithm 2: MRI Dataset Preprocessing

Input: M ←MRI data matrix Output: Mpro. 1. Image Quality Enhancement       M_enhanced ← EnhanceQuality(M)  // Correct spatial distortions and enhance contrast 2. Skull Stripping       M_brain ← StripSkull(M_enhanced) // Remove non-brain tissues 3. Normalization       M_normalized ← Normalize(M_brain) // Standardize intensity across images 4. Spatial Normalization       M_standardized ← AlignToTemplate(M_normalized) // Align images to a standard brain template 5. Smoothing       M_smoothed ← Smooth(M_standardized)       // Apply Gaussian smoothing filter 6. Output Processed Data       M_pro. ← M_smoothed         // Final processed MRI data

As far as the MRI data preprocessing is concerned, in the very first attempt, image quality enhancement was performed on each MRI, which included correcting for spatial distortions, aligning sequential images, and enhancing contrast to improve the clarity of the brain structures. After this, the skull stripping process [38] removes the non-brain tissues, like the skull and skin, from the MRI images, which are crucial for focusing the analysis on brain tissue. To standardize the intensity of the MRI images to a common scale, an adopted normalization process [38] has been applied that helps in comparing images across different subjects. However, spatial normalization aligns the images to a standard brain template and ensures that anatomical regions are consistently located across different subjects. In the end, a Gaussian filter [38] is applied to reduce noise and improve the signal-to-noise ratio.

3.3. Feature Extraction

Feature extraction is essential because it simplifies and condenses the vast and complex information provided by each modality into a more manageable and informative set of data. MRI scans offer detailed structural images of the brain, highlighting aspects such as brain volume and cortical thickness, while EEG recordings capture the brain’s dynamic electrical activity over time. By extracting key features from both MRI and EEG data, the hybrid model can leverage the complementary strengths of structural and functional brain insights. This synergy enhances the model’s performance. Moreover, feature extraction reduces the dimensionality of the data, which is crucial for effectively training machine learning algorithms without losing critical information. It helps in avoiding issues like overfitting and makes the computational process more efficient. In essence, feature extraction is a critical step in merging these two diverse data streams, paving the way for a more robust and accurate diagnosis of Parkinson’s Disease.

3.3.1. EEG Feature Extraction

The Flexible Analytic Wavelet Transform (FAWT) [39] is particularly effective for EEG feature extraction compared to other statistical methods due to its adaptive time-frequency analysis capabilities. Unlike the Fourier Transform, which only provides frequency information, or standard wavelet transforms with fixed time-frequency resolution, FAWT dynamically adjusts its analysis to match the local characteristics of EEG signals. Moreover, FAWT effectively handles non-stationary EEG signals, providing more accurate and meaningful feature extraction compared to methods that assume stationary signals. The range of features extracted, including instantaneous frequency and amplitude, provides a richer set of data for classifiers in machine learning models, potentially enhancing the accuracy of PD diagnosis.

In this research, For a particular EEG signal X(t), where t represents time. The continuous wavelet transform (CWT) of X(t) is given by the following equation:

$$Wx(a,b)=\int x(t)\psi a,b(t)dt$$

(1)

Here, ψ_a,b(t) is the wavelet function, a is the scale factor, and b is the translation factor.

The wavelet function ψ_a,b(t) is defined as follows:

$$\psi a,b\left(t\right)=\frac{1}{\sqrt{a}}\psi \left(\genfrac{}{}{0pt}{}{t-b}{a}\right)$$

(2)

ψ(t) is the mother wavelet, a prototype for generating other wavelets.

The adaptation of FAWT to the standard wavelet transform gives better signal characteristics. It performs this by allowing the wavelet function to change its shape (i.e., become more flexible) according to the signal’s local features. Mathematically, this adaptability can be incorporated by introducing a parameter θ(t) that modifies the wavelet function based on the signal’s local properties. The adaptive wavelet function can then be represented as follows:

$$\psi a,b,\theta \left(t\right)=\frac{1}{\sqrt{a}}\psi \theta \left(\genfrac{}{}{0pt}{}{t-b}{a}\right)$$

(3)

Here, ψθ(t) is the adaptive mother wavelet, which changes according to θ(t), a function of time. The FAWT of the EEG signal is computed to obtain a time-frequency representation. Features are then extracted from this representation, such as the energy, power, or frequency-specific components within specific time windows. These features capture the dynamic changes in the EEG signal, which are crucial for identifying PD. Some extracted features are shown in

Table 2. Extracted features from EEG.

Features	Values
Delta Band Power	10 μV2
Alpha Band Power	15 μV2
Beta Band Power	20 μV2
Theta/Beta Ratio	0.75
Spectral Entropy	0.85
Phase Locking Value	0.65
Cross-Frequency Coupling Strength	0.55

.

3.3.2. MRI Feature Extraction

For the extraction of MRI-related features, a Functional MRI (fMRI) based measure has been used. fMRI is a time series extraction for each voxel or Region of Interest (ROI). The extracted time series is represented by Ti(t) where Ti(t) is the BOLD signal over time t for the ith voxel or ROI. The functional connectivity has been computed by using the Pearson correlation coefficient between the time series of different ROIs or voxels such that the following holds:

$$C_{ij}=\frac{{\sum }_t{(T}_i\left(t\right)-{\mathrm{₸}}_i)-{(T}_j\left(t\right)-{\mathrm{₸}}_j)}{\sqrt{{\sum }_t{{(T}_i\left(t\right)-{\mathrm{₸}}_i)}^2-{\sum }_t{{(T}_j\left(t\right)-{\mathrm{₸}}_j)}^2}}$$

(4)

where C_ij is the correlation coefficient between ROI i and ROI j, Ti(t) and Tj(t) are the time series for ROIs i and j, respectively, and ₸_i, ₸_j are their mean values over time.

The functional connectivity graph has been formed by using graph metrics like degree, betweenness centrality, etc., to characterize the network, where nodes represent ROIs and edges represent the strength of connectivity (e.g., correlation). In the end, the General Linear Model (GLM) [39] models the BOLD signal in each voxel as a linear combination of explanatory variables (like task conditions) and confounds:

$$Ti\left(t\right)=\beta 0+\beta 1X1\left(t\right)+...+\beta nXn\left(t\right)+ϵi\left(t\right)$$

(5)

where $X1\left(t\right),...,Xn\left(t\right)$ are the explanatory variables, $\beta 0,...,\beta n$ are the coefficients, and ϵi(t) is the error term.

All the obtained features are standardized to the fMRI feature vector so that it is on a similar scale to the EEG feature vector. This step is crucial for effective data fusion and model training. Some extracted features from MRI are shown in

Table 3. Extracted features from MRI.

Features	Values
Resting-State Functional Connectivity	0.65
Regional Homogeneity (ReHo)	1.25
Low-Frequency Fluctuations (ALFF)	0.85
Regional Functional Connectivity Patterns	0.7
Task-Based Activation Patterns	1.2

.

3.4. Feature Fusion

The extracted features from both MRI and EEG datasets are concatenated to form a single, high-dimensional feature vector for each patient. In this work, the most prominent approach based on Principle Component Analysis (PCA) has been applied to future fusion. PCA is a statistical technique employed to diminish the dimensions of extensive datasets, maintaining the majority of the initial variance in the process.

The fused feature matrix X, where each row represents a patient and each column a feature (either from MRI or EEG). If there are n patients and p features, then X is an n × p matrix. PCA is sensitive to the variances of the initial variables.

$$X_{std}=\frac{X-\mu }{\sigma }$$

(6)

where μ is the mean and σ shows the standard deviation.

PCA starts with the computation of the covariance matrix Σ of the standardized data X_std. The covariance matrix expresses the covariance between each pair of features in the data such that the following holds:

$$\sum =\frac{1}{n-1}{X}_{std}^T{X}_{std}$$

(7)

PCA entails determining the eigenvectors and eigenvalues of the covariance matrix. It reveals the extent of variance in the data along the newly established feature axes. The eigenvectors are sorted by their eigenvalues in descending order. The first few eigenvectors (principal components) are selected based on the desired number of features or a set threshold for the amount of variance to be retained. Finally, the original standardized data X_std is projected onto these principal components to obtain the transformed feature set.

$$X_{pca}=X_{std}\times W$$

(8)

where W is the matrix containing the selected eigenvectors. Some eigenvalues and eigenvectors are shown in

Table 4. Eigenvalues and Eigenvector.

Eigenvalues	Eigenvector
0.92	[0.5,−0.3,0.1,0.7]
0.78	[−0.1,0.6,−0.7,0.2]
0.63	[0.7,0.2,0.4,−0.5]
0.55	[−0.2,−0.5,0.6,0.7]
0.41	[0.3,0.7,0.5,0.2]

.

The final feature vector is a combination of both fMRI and EEG features, ready to be input into a machine learning classifier for PD diagnosis. This vector encapsulates information about both the structural/functional aspects of the brain (from fMRI) and the electrical activity (from EEG). This integrated approach aims to leverage the strengths of both imaging techniques, offering a more comprehensive and nuanced view of the brain’s functioning, which is particularly beneficial in PD diagnosis processing.

3.5. Classification Using LightGBM

In this research, LightGBM [40], a highly efficient gradient-boosting framework, has been adopted for the classification of PD. The PD diagnosis features that are extracted from MRI and EEG data, the LightGBM model, can be finely tuned to differentiate between PD (positive) and non-PD (negative) cases. This binary classification task requires setting the objective parameter to binary.

Algorithm 3 gives a detailed description of the LightGBM-based PD classification. The dataset (x, y) is split into training and testing sets. The LightGBM model parameters are defined, including the objective, boosting type, and other parameters like the number of leaves, max depth, and fractions for feature and data sampling. The model is trained using the specified parameters and training data. The final Predictions are made on the test set and then converted to binary outputs.

Algorithm 3: LightGBM for PD Classification

Input: - X [features_matrix]  // Matrix of features extracted from MRI and EEG - y [labels_vector]   // Vector of labels (PD or non-PD) Output: - Model based Diagnosis 1. Data Preparation   X_train, X_test, y_train, y_test ← split_data(X, y, test_size) 2. Create LightGBM Dataset   train_data ← create_dataset(X_train, y_train) 3. Set LightGBM Parameters   params ← {     ‘objective’: ‘binary’,     ‘boosting_type’: [‘gbdt’, ‘goss’], ‘num_leaves’: set_number_of_leaves(),     ‘max_depth’: set_max_depth(), ‘min_data_in_leaf’: set_min_data_in_leaf(),     ‘feature_fraction’: set_feature_fraction(),‘bagging_fraction’: set_bagging_fraction()       } 4. Model Training   model ← lgb.train(params, train_data) 5. Model Predictions   y_pred ← model.predict(X_test) 6. Transform Probabilities to Binary Output (for binary classification)   y_pred_binary ← convert_to_binary(y_pred)

4. Experimental Results and Evaluation

This section provides an in-depth analysis of the performance evaluation of the proposed model, with each aspect elaborated in detail in the following subsections.

4.1. Dataset Description

Two distinct types of datasets have been used for the analysis of the proposed work.

Table 5. Dataset Description.

Dataset Name	Nature of Data	Age Group	Class Label
PD-DS-I	Multimodal EEG and MRI	Avg. 67 [Ranging from 21–89]	PD patient 530	Non-PD patient 250
PD-DS-II	Multimodal EEG and MRI	Avg. 50 [Ranging from 20–60]	PD patient 347	Non-PD patient 213
Web link of Dataset
Name	Address
PD-DS-I	https://shorturl.at/tuzGJ (accessed on 26 December 2023)
PD-DS-II	https://shorturl.at/qtHI5 (accessed on 26 December 2023)

shows the distinctive detail of each dataset. The first dataset, abbreviated as PD-DS-I, has been obtained from the Latin American Brain Health Institute (BrainLat) and has unveiled a significant multimodal neuroimaging dataset comprising 780 participants. This collection features 530 individuals diagnosed with PD and 250 healthy controls (HCs). Averaging an age of 62.7 years and ranging from 21 to 89 years, this dataset was gathered as part of a metacentric initiative across five countries in Latin America. BrainLat marks the first regional compilation that includes a variety of assessments specifically focused on PD patients.

The very next dataset (PD-DS-II) is a collection of anatomical MRI that directly identifies neuronal loss in PD. Exploring changes in functional connectivity (FC) offers a promising route to develop non-invasive and radiation-free neuroimaging markers for the disease. The PD-DS-II utilizes data from two sources: the Neurocon dataset, which includes 127 PD patients and 16 age-matched healthy controls, and the Tao Wu dataset, comprising 220 PD patients and 20 age-matched controls. Both datasets provide T1-weighted and resting-state scans.

4.2. Performance Measures

To assess the stability and reliability of the proposed method, we employ a 10-fold cross-validation (CV) procedure. This approach mitigates the risks of both underfitting and overfitting and helps reduce the variance when analyzing results. In this process, the feature vector is split into ten random subsets. In each fold, nine subsets are used for training, while the remaining one is used for testing. After 10 iterations (folds), the training algorithm’s performance measures are averaged. The proposed model learning rate is 0.001, with a batch size of 32 and training over around 50 epochs. A dropout rate of 0.5 is often applied for regularization, along with weight initialization.

The outcome of this process is a confusion matrix that provides values for true negative (TN), true positive (TP), false negative (FN), and false positive (FP). Here, TP and TN represent correct predictions, where Parkinson’s patients and healthy controls are accurately identified. Conversely, FP and FN indicate incorrect predictions, marking individuals as having Parkinson’s when they do not (FP) and vice versa (FN).

Performance metrics are crucial for evaluating the effectiveness of our study in detecting PD patients. Additionally, we measure computational time to assess the real-time applicability of the classifier. Each of these measures is explained in brief as follows:

• Accuracy: This metric determines the overall effectiveness of a classifier in correctly distinguishing between negative and positive cases. It reflects the proportion of true results in relation to all the cases examined.

$$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$$

(9)

• Sensitivity: This metric assesses the classifier’s ability to correctly identify positive cases. It indicates the proportion of actual positives (true positives) that are correctly identified by the classifier, reflecting its effectiveness in detecting cases where the condition is present.

$$Sencitivity=\frac{TP}{TP+FN}$$

(10)

• Specificity: This metric measures the classifier’s ability to accurately identify negative cases. It indicates the proportion of actual negatives (true negatives) that are correctly identified, reflecting the classifier’s effectiveness in recognizing cases where the condition is absent.

$$Specificity=\frac{TN}{TN+FN}$$

(11)

• Mathew’s Correlation Coefficient (MCC): This metric provides a balanced measure that evaluates the quality of binary classifications, taking into account both the positive and negative classes. MCC considers true and false positives and negatives, offering a comprehensive assessment of a classifier’s performance, particularly in situations where class imbalances are present. It ranges from −1 to +1, where +1 indicates a perfect prediction, 0 denotes random prediction, and −1 represents complete disagreement between observation and prediction.

$$MCC=\frac{\left(TP+TN+FP+FN\right)-(FP+FN)}{\sqrt{(TP+FN)(TP+FP)(TN+FN)(TN+FP)}}$$

(12)

• Area under the Receiver Operating Characteristics Curve (AUC): The Area under the Curve (AUC) of the Receiver Operating Characteristics (ROC) is a statistical metric for assessing the effectiveness of a binary classifier. It quantifies the likelihood that the model correctly ranks a randomly selected positive sample above a negative one. The superior performance of the model is indicated by a higher AUC-ROC score, with values nearing 1 denoting exceptional predictive precision.
• ANOVA: ANOVA works by comparing the variance between groups (the variation between the group means) with the variance within groups (the variation within each group). If the between-group variance is significantly greater than the within-group variance, it suggests that there are statistically significant differences among the group means.
• Kappa Statistic: This metric evaluates the agreement between the classification results and the true values. It compares the observed accuracy with what would be expected by chance. Mathematically, the Kappa statistic is calculated as follows:

$$Kappa Statisctics=\frac{2\times (TP\times TN-FN\times FP)}{(TP+FP)\times (FP+TN)+(TP+FN)\times (FN+TN)}$$

(13)

4.3. Baseline Techniques

Due to the novelty and multimodal nature of the proposed work, very little research has been found in this domain. However, three different baseline methods have been selected for the comparative analysis of the proposed work.

• Makarious et al. [15]: An automated machine learning (ML) system was developed to analyze multimodal data from the Parkinson’s Progression Markers Initiative (PPMI). Following the selection of the most effective algorithm, the complete PPMI dataset was utilized to fine-tune their chosen model. Subsequently, their model underwent validation using the PD Biomarker Program (PDBP) dataset. The preliminary results of our model demonstrated an area under the curve (AUC) of 89.72% in diagnosing PD.
• Papadopoulos et al. [41]: A deep learning framework developed to simultaneously analyze multimodal data, enabling concurrent predictions in three key areas: tremor severity, fine-motor skill impairment, and the likelihood or presence of Parkinson’s disease (PD). This advanced framework integrates various data types to provide a comprehensive assessment of these interconnected health aspects.

Building upon the aforementioned methods, our study also incorporates some of the single-model machine learning techniques for a comprehensive comparative analysis. The specifics of these additional models are detailed as follows:

• Al-Khasawneh et al. [42]: Proposed an Artificial Intelligence-based approach for the effective diagnosis of PD using EEG signals. Their study centers on utilizing human bio-signals as a means for early detection of PD.
• Nan Xo et al. [43]: Designed a biomarker that was developed using Topological Machine Learning applied to Resting-state Functional Magnetic Resonance Imaging (rs-fMRI). This work involved the creation of a spatial-temporal dimension reduction technique for rs-fMRI, specifically tailored for the automated diagnosis of PD.

4.4. Results

The initial experiment was conducted to validate the efficacy of the proposed methodology in diagnosing PD. This was achieved by assessing key performance metrics such as specificity, accuracy, sensitivity, and F-score. The outcomes, as illustrated in Figure 6, showcase the statistical values derived from applying the proposed approach to two datasets, PD-DS-I and PD-DS-II. The results from this demonstration clearly indicate that the proposed method consistently achieved outstanding performance across both datasets. The analysis of variance (ANOVA) was conducted, yielding an F-score of 0.348 and a corresponding p-value of 0.794. As the p-value exceeds the significance threshold, we fail to reject the null hypothesis. Thus, there is no statistically significant difference observed between the groups.

Figure 7 displays the confusion matrix for both datasets, utilized in calculating various performance metrics, and highlights the superior performance of the proposed technique. Additionally, the efficacy of the LightGBM classifier is further validated through the Area under the Receiver Operating Characteristics Curve (ROC-AUC).

The ROC-AUC graphs for both PD-DS-I and PD-DS-II are illustrated in Figure 7, where the AUC values are recorded at 0.991 and 0.959, respectively. These high AUC values reinforce the effectiveness of the proposed approach in accurately distinguishing PD patients from healthy individuals. This accuracy is achieved by utilizing multimodal features extracted from EEG signals and MRI, demonstrating the proposed model’s capability in PD diagnosis. The Confusion matrix is shown in Figure 8.

In a subsequent experiment, the performance of our proposed model was benchmarked against the studies by Makarious et al. and Papadopoulos et al. This comparative analysis, as detailed in

Table 6. Comparative analysis of proposed work with machine learning models.

Model	Sensitivity	Specificity	MCC	AUC	Kappa Statistics
PD-DS-I Dataset
Proposed Approach	97.02	97.28	0.97	0.96	0.97
Al-Khasawneh et al.	86.63	85.45	0.86	0.86	0.85
Nan Xo et al.	92.36	92.63	0.92	0.91	0.91
PD-DS-II Dataset
Proposed Approach	96.13	96.36	0.96	0.95	0.96
Al-Khasawneh et al.	89.63	88.45	0.89	0.89	0.89
Nan Xo et al.	91.23	91.78	0.91	0.90	0.91

, unequivocally demonstrates that our approach significantly outperformed the baseline studies. Specifically, the proposed model exhibited an accuracy improvement of 11% on PD-DS-I and 10.2% on PD-DS-II, respectively, showcasing its superior diagnostic capabilities in PD detection. The ROC curves for PD-DS-1 and PD-DS-II are in Figure 9.

Figure 10 presents a comprehensive evaluation of various classifiers, focusing on classification accuracy, specificity, sensitivity, MCC, ROC, and the Kappa statistic. This analysis reveals that the proposed model achieves the highest overall classification performance. The proposed model’s accuracy, sensitivity, and specificity, as obtained after implementing 10-fold cross-validation, are exceptionally high at 97.17%, 96.58%, and 96.82%, respectively.

Moreover, the AUC value for the proposed work is 0.96, categorizing it as an almost perfect classification method. In contrast, while the Al-Khasawneh et al. classifier shows a slightly higher AUC of 0.86, its performance in other areas, such as classification specificity (85.45%), sensitivity (86.63%), and MCC (0.86), is lower compared to proposed work. Additionally, the Kappa statistic for this classifier is noted at 0.85, the highest among the classifiers reviewed, indicating near-perfect classification stability and a high degree of alignment between the classification and actual values. This suggests that the proposed work almost accurately classifies all test instances. For ANOVA, we ensured normality, homogeneity of variance, and independence of observations. Similarly, for the Kappa statistic, we confirmed the independence of raters, the appropriate scale of measurement, and the randomness of sampling.

5. Discussion

In this research, we have developed a multimodal machine learning approach for Parkinson’s disease diagnosis using EEG and MRI data by utilizing FAWT, fMPRI, PCA, and LightGBM. The innovation of this study lies in the fusion of different modality datasets for discerning PD patients from healthy individuals. While many neuroimaging techniques are available for PD detection, they often require interpretation by experienced neurophysiologists and are prone to errors due to manual analysis, in addition to being costly.

PD, a severe neurodegenerative disorder, diminishes dopamine levels in the brain, impacting the electrical potentials of neurons. These changes are closely associated with neurophysiological signals, specifically EEGs and MRI, which are recorded directly via scalp electrodes. Therefore, this work focuses on utilizing EEG signals as well as MRI for PD detection. EEG and MRI are regarded as crucial for the early and effective diagnosis of PD, offering low error rates, and are heavily relied upon by healthcare professionals in clinical decision-making.

The integration of EEG and MRI data for Parkinson’s disease diagnosis can offer more accurate and early detection. This approach may enable personalized treatment strategies, optimizing medication regimens, or timing of interventions. Additionally, it could provide insights into the underlying neural mechanisms of PD progression, potentially leading to the development of targeted therapies and improving patient outcomes.

We further validated the performance of our framework by comparing it with other cutting-edge studies. Figure 10 and Table 6 present a comparison of accuracy with other contemporary works using dataset-I, dataset-II, and various state-of-the-art datasets, providing a comprehensive perspective of our analysis. Consequently, this discussion and comparative analysis strongly support that the proposed approach achieved better results as compared to the baselines.

Despite its high detection rate in differentiating PD from HC using FAWT, fMPRI, PCA, and LightGBM, our methodology has certain limitations, which are as follows:

• The approach was formulated and evaluated using a small sample size from two datasets. To enhance the reliability of Parkinson’s disease (PD) detection, it is recommended that the algorithms undergo testing with a more extensive sample base and across datasets from various regions.
• The research was exclusively concentrated on a single nonlinear attribute, entropy, for diagnosing PD. Subsequent studies could investigate an array of nonlinear attributes and their synergies to derive more sophisticated features from EEG and MRI data for PD diagnosis.
• One significant limitation of the study could stem from the variability in data quality and standardization across EEG and MRI acquisitions. Ensuring consistency in equipment, protocols, and participant factors across multiple sites is crucial but challenging. Additionally, limited sample sizes or homogeneity in participant demographics may restrict the generalizability of findings. Furthermore, integrating EEG and MRI data requires sophisticated analytical techniques, potentially introducing interpretation challenges and subjectivity, emphasizing the importance of transparent analysis methods and independent replication studies for validation.

6. Conclusions

Parkinson’s disease is a neurodegenerative disorder characterized by symptoms that often overlap with other neurological conditions, highlighting the need for early and precise diagnosis to effectively manage treatment and improve a patient’s quality of life. This study introduces an innovative approach for PD diagnosis, utilizing a multimodal framework that integrates Electroencephalography (EEG) and Magnetic Resonance Imaging (MRI) data. Our approach focuses on harnessing the distinct and complementary strengths of EEG and MRI to enhance diagnostic accuracy, particularly in the early stages of PD. The research employs a robust algorithmic strategy, including LightGBM and other advanced machine learning techniques, to analyze complex neurological patterns. Key steps in our methodology include preprocessing and feature extraction from both EEG and MRI, followed by fusion using Principal Component Analysis (PCA) for dimensionality reduction. The fused dataset is then evaluated using a LightGBM model, rigorously validated through a 10-fold cross-validation process to ensure stability and reliability. The results demonstrate high levels of accuracy, sensitivity, and specificity in PD classification, which surpass traditional methods. This approach not only offers a more comprehensive view of PD-related neurophysiological and neuroanatomical changes but also represents a significant advancement in the field of PD diagnosis.

Future work could involve standardizing data acquisition protocols, expanding sample sizes, and refining analytical methods to address limitations. Additionally, exploring novel biomarkers or imaging modalities may enhance diagnostic accuracy and deepen the understanding of Parkinson’s disease progression. These efforts could significantly advance diagnosis and management strategies for Parkinson’s disease.