Multiple Inputs and Mixed Data for Alzheimer’s Disease Classification Based on 3D Vision Transformer
Abstract
:1. Introduction
- Data Preparation (Phase 1): This phase involves splitting the dataset into training, validation, and test sets. Additionally, raw MRI volumes undergo skull-stripping and registration preprocessing to create refined datasets.
- Instance Selection (Phase 2): This phase involves selecting slice instances that contain the regions of interest (ROIs) and calculating the centroid position (x, y) based on the statistical mode, which identifies the most frequent values for accurate ROI localization.
- Model Validation (Phase 3): This phase involves batch data generation, training, and testing diverse classification models based on transformers. It also includes integrating multiple inputs by combining the categorical, numerical, and image data from different ROIs to enhance the model performance.
- Introducing a 3D Vision Transformer model that processes consecutive slices together, capturing the feature dimensions and spatial information through self-attention mechanisms. This model significantly improves Alzheimer’s Disease diagnosis and classification by overcoming the limitations of 2D independent analysis.
- Fusing multiple 3D ROI imaging data inputs from Alzheimer’s Disease-affected areas helps to overcome the limitation of focusing on only one or a few brain regions. This approach enhances the model performance and robustness in Alzheimer’s Disease diagnosis and classification.
- Integrating mixed data from demographic factors, cognitive assessments, and brain imaging is essential because no single test can definitively diagnose Alzheimer’s. This multifaceted approach improves the robustness and reliability of the diagnostic models.
2. Related Works
3. Materials and Methods
3.1. Datasets
3.2. Proposed Methodology
3.2.1. Dataset Preparation
Volume Dataset Building
Data Preprocessing
- Numerical Data: Min–max scaled to the range [0, 1] before training.
- Categorical Data: One-hot encoded, ensuring the resulting integer vectors were in the range [0, 1].
- Image Data:
- –
- Skull-Stripping: Raw volumes selected in the previous “Volume Dataset Building” step from ADNI, AIBL, or OASIS repositories underwent skull-stripping to isolate the brain from non-brain tissues such as skin, fat, muscle, neck, and eyeballs.
- –
- Registration: Skull-stripped dataset volumes were registered with the MNI152 T1 template MRI scan, ensuring uniformity in shape, position, and alignment. The resulting scans have dimensions of 182 × 218 × 182 and a resolution of 1 mm.
- –
- Scaling: Image data were scaled to the range [0, 1].
3.2.2. Instance Dataset Building
3.2.3. Data Generation
3.2.4. Model Building
Multiple-Input Network
- Multiple-Input Network: To build our Multiple Inputs and Mixed Data network, we will need two branches, as shown in Figure 2:
- –
- The first branch will be a simple Multi-layer Perceptron (MLP) designed to handle the categorical/numerical inputs.
- –
- The second branch will be a Swin Transformer to operate over the Magnetic Resonance image data.
- –
- These branches will then be concatenated together to form the final multiple-input model.
- 3D Vision Transformer: This study uses ViViT: A Video Vision Transformer [31], a pure Transformer-based model, to classify AD MRI 3D ROIs. Figure 3 shows the Tubelet embedding method used to extract non-overlapping spatio-temporal “tubes” from the input volume and linearly project them to . This method extends ViT’s embedding to 3D and corresponds to a 3D convolution. For a tubelet of dimension , tokens are extracted from the temporal, height, and width dimensions, intuitively fusing spatio-temporal information during tokenization [31].
Hyperparameter Optimization
Model Performance Evaluation
4. Experimental Setup
4.1. Experiments
4.1.1. Mixed Data
4.1.2. Performance Comparison
4.2. Hyperparameter Tuning
4.3. Data Analysis
- (null hypothesis): (It implies that the mean accuracy of all models is equal).
- (alternative hypothesis): It states that at least one model accuracy mean will differ from the rest.
4.4. Implementation Details
5. Results and Discussion
5.1. Experiment I (Mixed Data)
5.2. Experiment II (Performance Comparison)
6. Limitations
7. Conclusions and Future Work
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- World Health Organization. Dementia. Available online: https://www.who.int/news-room/fact-sheets/detail/dementia (accessed on 15 November 2023).
- National Institute on Aging. What Happens to the Brain in Alzheimer’s Disease? Available online: https://www.nia.nih.gov/health/what-happens-brain-alzheimers-disease (accessed on 15 November 2023).
- Alzheimer’s Society. Dementia Symptoms and Areas of the Brain. Available online: https://www.alzheimers.org.uk/about-dementia/symptoms-and-diagnosis/how-dementia-progresses/symptoms-brain#content-start (accessed on 15 November 2023).
- Yu, J.; Lee, T.M. Verbal memory and hippocampal volume predict subsequent fornix microstructure in those at risk for Alzheimer’s disease. Brain Imaging Behav. 2020, 14, 2311–2322. [Google Scholar] [CrossRef] [PubMed]
- Alzheimer’s Association. Medical Tests for Diagnosing Alzheimer’s. Available online: https://www.alz.org/alzheimers-dementia/diagnosis/medical_tests (accessed on 10 December 2023).
- Wen, J.; Thibeau-Sutre, E.; Diaz-Melo, M.; Samper-González, J.; Routier, A.; Bottani, S.; Dormont, D.; Durrleman, S.; Burgos, N.; Colliot, O. Overview of classification of Alzheimer’s disease. Med. Image Anal. 2020, 63. [Google Scholar] [CrossRef] [PubMed]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.; Polosukhin, I. Attention Is All You Need. In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 4–9 December 2017. [Google Scholar]
- Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An Image Is Worth 16 × 16 Words: Transformers for Image Recognition At Scale. In Proceedings of the ICLR 2021—9th International Conference on Learning Representations, Vienna, Austria, 4 May 2021. [Google Scholar]
- Zhu, D.; Wang, D. Journal of Radiation Research and Applied Sciences Transformers and their application to medical image processing: A review. J. Radiat. Res. Appl. Sci. 2023, 16, 100680. [Google Scholar] [CrossRef]
- Zhang, L.; Chen, Z.; Zou, B.; Gao, Y. Polarimetric SAR Terrain Classification Using 3D Convolutional Neural Network. In Proceedings of the IGARSS 2018—2018 IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain, 22–27 July 2018; pp. 4551–4554. [Google Scholar] [CrossRef]
- Liu, L.; Liu, S.; Zhang, L.; To, X.V.; Nasrallah, F.; Chandra, S.S. Cascaded Multi-Modal Mixing Transformers for Alzheimer’s Disease Classification with Incomplete Data. NeuroImage 2023, 277, 120267. [Google Scholar] [CrossRef] [PubMed]
- Lyu, Y.; Yu, X.; Zhu, D.; Zhang, L. Classification of Alzheimer’s Disease via Vision Transformer: Classification of Alzheimer’s Disease via Vision Transformer. In ACM International Conference Proceeding Series, Proceedings of the 15th International Conference on Pervasive Technologies Related to Assistive Environments, Corfu, Greece, 29 June–1 July 2022; ACM: New York, NY, USA, 2022; pp. 463–468. [Google Scholar] [CrossRef]
- Mora-Rubio, A.; Bravo-Ortíz, M.A.; Arredondo, S.Q.; Torres, J.M.S.; Ruz, G.A.; Tabares-Soto, R. Classification of Alzheimer’s disease stages from magnetic resonance images using deep learning. PeerJ Comput. Sci. 2023, 9, e1490. [Google Scholar] [CrossRef] [PubMed]
- Altay, F.; Sánchez, G.R.; James, Y.; Faraone, S.V.; Velipasalar, S.; Salekin, A. Preclinical Stage Alzheimer’s Disease Detection Using Magnetic Resonance Image Scans. Proc. AAAI Conf. Artif. Intell. 2021, 35, 15088–15097. [Google Scholar] [CrossRef]
- Zhang, Z.; Khalvati, F. Introducing Vision Transformer for Alzheimer’s Disease classification task with 3D input. arXiv 2022, arXiv:2210.01177. [Google Scholar] [CrossRef]
- Huang, Y.; Li, W. Resizer Swin Transformer-Based Classification Using sMRI for Alzheimer’s Disease. Appl. Sci. 2023, 13, 9310. [Google Scholar] [CrossRef]
- Xin, J.; Wang, A.; Guo, R.; Liu, W.; Tang, X. CNN and swin-transformer based efficient model for Alzheimer’s disease diagnosis with sMRI. Biomed. Signal Process. Control 2023, 86. [Google Scholar] [CrossRef]
- Hu, Z.; Li, Y.; Wang, Z.; Zhang, S.; Hou, W. Conv-Swinformer: Integration of CNN and shift window attention for Alzheimer’s disease classification. Comput. Biol. Med. 2023, 164, 107304. [Google Scholar] [CrossRef] [PubMed]
- Li, C.; Cui, Y.; Luo, N.; Liu, Y.; Bourgeat, P.; Fripp, J.; Jiang, T. Trans-ResNet: Integrating Transformers and CNNs for Alzheimer’s disease classification. In Proceedings of the 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), Kolkata, India, 28–31 March 2022; pp. 1–5. [Google Scholar] [CrossRef]
- Zaabi, M.; Smaoui, N.; Derbel, H.; Hariri, W. Alzheimer’s disease detection using convolutional neural networks and transfer learning based methods. In Proceedings of the 2020 17th International Multi-Conference on Systems, Signals & Devices (SSD), Monastir, Tunisia, 20–23 July 2020; pp. 939–943. [Google Scholar] [CrossRef]
- Bae, J.B.; Lee, S.; Jung, W.; Park, S.; Kim, W.; Oh, H.; Han, J.W.; Kim, G.E.; Kim, J.S.; Kim, J.H.; et al. Identification of Alzheimer’s disease using a convolutional neural network model based on T1-weighted magnetic resonance imaging. Sci. Rep. 2020, 10, 1–10. [Google Scholar] [CrossRef] [PubMed]
- Ahmed, S.; Kim, B.C.; Lee, K.H.; Jung, H.Y. Ensemble of ROI-based convolutional neural network classifiers for staging the Alzheimer disease spectrum from magnetic resonance imaging. PLoS ONE 2020, 15, e0242712. [Google Scholar] [CrossRef] [PubMed]
- Pan, D.; Luo, G.; Zeng, A.; Zou, C.; Liang, H.; Wang, J.; Zhang, T.; Yang, B. The Alzheimer’s Disease Neuroimaging Initiative. Adaptive 3DCNN-Based Interpretable Ensemble Model for Early Diagnosis of Alzheimer’s Disease. IEEE Trans. Comput. Soc. Syst. 2022, 11, 247–266. [Google Scholar] [CrossRef]
- Poloni, K.M.; Ferrari, R.J. Automated detection, selection and classification of hippocampal landmark points for the diagnosis of Alzheimer’s disease. Comput. Methods Prog. Biomed. 2022, 214, 106581. [Google Scholar] [CrossRef] [PubMed]
- Aghaei, A.; Moghaddam, M.E. Smart ROI Detection for Alzheimer’s disease prediction using explainable AI. arXiv 2023, arXiv:2303.10401. [Google Scholar] [CrossRef]
- Castro-Silva, A.J.; Moreno-Garcia., M.; Guachi-Guachi., L.; Peluffo-Ordoñez., D.H. Instance Selection Framework for Alzheimer’s Disease Classification Using Multiple Regions of Interest and Atlas Integration. In Proceedings of the 13th International Conference on Pattern Recognition Applications and Methods—ICPRAM. INSTICC, Rome, Italy, 24–26 February 2024; SciTePress: Setubal, Portugal, 2024; pp. 453–460. [Google Scholar] [CrossRef]
- Alzheimer’s Disease Neuroimaging Initiative. ADNI. Available online: http://adni.loni.usc.edu (accessed on 15 November 2023).
- Australian Imaging, Biomarker & Lifestyle Flagship Study of Ageing. AIBL. Available online: https://aibl.csiro.au (accessed on 15 November 2023).
- Open Access Series of Imaging Studies. OASIS. Available online: http://www.oasis-brains.org (accessed on 15 November 2023).
- Hughes, C.P.; Berg, L.; Danziger, W.; Coben, L.A.; Martin, R.L. A New Clinical Scale for the Staging of Dementia. Br. J. Psychiatry 1982, 140, 566–572. [Google Scholar] [CrossRef] [PubMed]
- Arnab, A.; Dehghani, M.; Heigold, G.; Sun, C.; Lučić, M.; Schmid, C. ViViT: A Video Vision Transformer. arXiv 2021, arXiv:2103.15691. [Google Scholar] [CrossRef]
- Li, L.; Jamieson, K.; DeSalvo, G.; Rostamizadeh, A.; Talwalkar, A. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. J. Mach. Learn. Res. 2018, 18, 1–52. [Google Scholar]
Dataset | Class | Subjects | Age | Gender F/M | Total Subjects |
---|---|---|---|---|---|
ADNI | CN | 70 | 34/36 | 140 | |
AD | 70 | 31/39 | |||
AIBL | CN | 70 | 37/33 | 140 | |
AD | 70 | 43/27 | |||
OASIS | CN | 70 | 39/31 | 140 | |
AD | 70 | 34/36 | |||
MERGED | CN | 210 | 110/100 | 420 | |
AD | 210 | 108/102 |
Hyperparameter | Value | Description |
---|---|---|
Dataset | Merged | 70 Subjects each dataset (ADNI + AIBL + OASIS). |
Slice Number | 25 | Number of selected slices from ROI. |
Image Size | 32 × 32 | Image size in pixels. |
Number of channels | 3 | Number of channels (3 = RGB, 1 = Gray scale). |
Learning rate | LRS | LearningRateSchedule exponential decay. |
Decay steps | 100,000 | Drops the learning rate by a factor. |
Decay rate | The rate at which the learning rate is decayed. | |
Optimizer Name | Adam | The optimizer name. |
Initial Learning Rate | The initial learning rate. | |
Dropout | The rate at which the learning rate is decayed. | |
Batch Size | 6 | The number of instances per batch. |
Epochs | 250 | The number of epochs. |
Only-Image Data | Mixed Data | |||
---|---|---|---|---|
ROI | Left | Right | Left | Right |
Entorhinal Cortex | ||||
Fornix | ||||
Frontal Lobe | ||||
Hippocampus | ||||
Parietal Lobe | ||||
Temporal Lobe | ||||
Mean |
Study | Dataset | Model | Accuracy |
---|---|---|---|
[19] | ADNI | Trans-ResNet | |
AIBL | |||
[12] | ADNI | Vision Transformer | |
[18] | ADNI, OASIS | 2D CNN+Transformer | |
[14] | OASIS | Vision Transformer | |
[17] | ADNI, AIBL | CNN+Swin Transformer | |
[16] | ADNI+AIBL | Swin Transformer | |
[13] | ADNI+OASIS | Vision Transformer | |
Our Methodology | Merged (ADNI + AIBL + OASIS) | Multiple Inputs and Mixed Data | |
Multiple ROIs | 3D Vision Transformer | ||
Single ROI | Entorhinal Cortex-Left | ||
Fornix-Right | |||
Frontal Lobe-Left | |||
Hippocampus-Left | |||
Parietal Lobe-Right | |||
Temporal Lobe-Left |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Castro-Silva, J.A.; Moreno-García, M.N.; Peluffo-Ordóñez, D.H. Multiple Inputs and Mixed Data for Alzheimer’s Disease Classification Based on 3D Vision Transformer. Mathematics 2024, 12, 2720. https://doi.org/10.3390/math12172720
Castro-Silva JA, Moreno-García MN, Peluffo-Ordóñez DH. Multiple Inputs and Mixed Data for Alzheimer’s Disease Classification Based on 3D Vision Transformer. Mathematics. 2024; 12(17):2720. https://doi.org/10.3390/math12172720
Chicago/Turabian StyleCastro-Silva, Juan A., María N. Moreno-García, and Diego H. Peluffo-Ordóñez. 2024. "Multiple Inputs and Mixed Data for Alzheimer’s Disease Classification Based on 3D Vision Transformer" Mathematics 12, no. 17: 2720. https://doi.org/10.3390/math12172720