Multimodal Deep Learning in Early Autism Detection—Recent Advances and Challenges ^†

Abstract

Autism spectrum disorder (ASD) is a global concern, with a prevalence rate of approximately 1 in 36 children according to estimates from the Centers for Disease Control and Prevention (CDC). Diagnosing ASD poses challenges due to the absence of a definitive medical test. Instead, doctors rely on a comprehensive evaluation of a child’s developmental background and behavior to reach a diagnosis. Although ASD can occasionally be identified in children aged 18 months or younger, a reliable diagnosis by an experienced professional is typically made by the age of two. Early detection of ASD is crucial for timely interventions and improved outcomes. In recent years, the field of early diagnosis of ASD has been greatly impacted by the emergence of deep learning models, which have brought about a revolution by greatly improving the accuracy and efficiency of ASD detection. The objective of this review paper is to examine the recent progress in early ASD detection through the utilization of multimodal deep learning techniques. The analysis revealed that integrating multiple modalities, including neuroimaging, genetics, and behavioral data, is key to achieving higher accuracy in early ASD detection. It is also evident that, while neuroimaging data holds promise and has the potential to contribute to higher accuracy in ASD detection, it is most effective when combined with other modalities. Deep learning models, with their ability to analyze complex patterns and extract meaningful features from large datasets, offer great promise in addressing the challenge of early ASD detection. Among various models used, CNN, DNN, GCN, and hybrid models have exhibited encouraging outcomes in the early detection of ASD. The review highlights the significance of developing accurate and easily accessible tools that utilize artificial intelligence (AI) to aid healthcare professionals, parents, and caregivers in early ASD symptom recognition. These tools would enable timely interventions, ensuring that necessary actions are taken during the initial stages.

1. Introduction

Autism spectrum disorder (ASD) is a developmental condition affecting 1–2% of children worldwide, causing social interaction challenges, communication difficulties, and repetitive behaviors. Figure 1 shows the issues faced by children with ASD. Genetics and environmental factors significantly impact its development. Advances in diagnosis provide hope for improved outcomes [1,2,3,4]. ASD individuals face challenges such as social interaction difficulties, communication issues, repetitive behaviors, and sensory sensitivities [5,6,7,8]. The assessment and diagnosis of ASD largely rely on traditional clinical evaluations that have been utilized for several decades, as shown in Figure 2. Deep learning techniques are increasingly used for ASD detection, and integrate data from various sources to enhance accuracy [9]. The choice of modalities depends on available data and research goals [10]. Deep learning (DL) methods are increasingly used in early ASD detection and for analyzing data from neuroimaging, behavioral observations, and speech [11]. This enhances diagnostic accuracy and timeliness, potentially improving outcomes [12]. fMRI and sMRI play vital roles in accurate diagnosis [13]. AI-based CAS employs both ML and DL approaches, but DL techniques are underutilized [14,15,16]. Advancements in ASD diagnostics use DL models, combining neuroimaging methods with ML and DL, to identify early biological markers [17,18,19]. Lightweight CNN models show high accuracy, precision, and F1 score. Challenges include data quality, interpretability, generalizability, and ethical considerations [14,20].

2. Methodology

This systematic review uses PRISMA methodology to analyze early ASD detection advancements using multimodal DL techniques. It employs structured research methods, including clear questions, eligibility criteria, literature search, systematic screening, and data extraction. The review discusses implications and challenges, considering strengths and limitations. A systematic search approach was used to evaluate each article’s suitability to address the research questions. In this review, databases like Google Scholar, PubMed, and IEEE were used to acquire the current study of neurodevelopmental disorders in children using machine learning techniques. Relevant articles were shortlisted using keywords like “Deep Learning” and “Autism Spectrum Disorder”. Figure 3 shows the flow and the number of articles identified through different sources, which focused on publications from 2019–2023. After thorough examination of titles, abstracts, and full contents, 35 articles were selected for further analysis.

3. Data Synthesis and Analysis

Several studies have delved into recent advancements in early ASD detection through multimodal DL techniques using neuroimaging and non-neuroimaging data. Khodatars et al. [9] explored DL and AI’s role in precise ASD diagnosis and rehabilitation, offering insights for future research directions. de Belen et al. [21] highlighted the effectiveness of computer vision analysis in quantifying ASD markers, benefiting diagnosis and therapy. Feng et al. [22] assessed ML and DL methods for fMRI-based ASD classification and recognition, discussing performance and challenges in early diagnosis. Haweel et al. [23] investigated the potential of deep learning techniques using TfMRI for early ASD diagnosis and the identification of autism biomarkers.

Table 1. Summary of multimodal DL techniques in ASD detection as reported in previous studies.

Author	Model Used	Feature Used	Accuracy	Modality Used
Ming Li [24]	CNN+RNN	Open SMILE and CQT spectrogram	88.1%	Behavior signal, speech
Yang et al. [25]	ASSDL	Neuroimaging	98.2%	fMRI
Huang et al. [26]	DBN	Graph-based feature selection (GBFS)	76.4%	fMRI from ABIDE
Pan et al. [27]	GCN	Brain imaging	87.62%	fMRI from ABIDE I
Niu et al. [28]	DANN	Multi scale brain functional connectom	73.2%	rs-fMRI and PC data from ABIDE
Ahmed et al. [29]	Mobile Net Xception InceptionV3	Facial features	95% 94% 89%	Facial images
Saputra et al. [30]	CNN	rs-fMRI and task-fMRI BOLD signals, and aberrations in brain disorders	89.58%	Brain MRI, clinical and behavioural markers, electroencephalography indices
Liao et al. [31]	CNN	Features fusion	87.50%	Eye fixation, facial expression, and EEG
Sharif, and Khan, [32]	CNN	Corpuscallosum	55.93%	Neuroimaging data, EEG, speech, Kinesthetic
Epalle et al. [33]	MISO-DNN	Features fusion	79.13%	MRI
Ke et al. [34]	2D/3D CNN	Spatial transformer network (STN) and classification activation mapping (CAM)	89%	MRI
Almuqhim and Saeed, [35]	ASD-SAENet	Sparse autoencoder (SAE)	70.8%	fMRI
Lee et al. [36]	BLSTM	eGeMAPS speech feature	68.18%	ADOS-2, ADI-R, BeDevel-I, BeDevel-P, K-CARS, SCQ, and SRS
Rahman and Subashini [37]	DNN	Feature fusion	97.18%	QCHAT and QCHAT-10
Saranya and Anandan, [38]	DEAF	Multimodal features	96.5	Facial fusion emotions and human gait sequences
Shao et al. [39]	GCN	Deep features	79.5%	fMRI from ABIDE
Israr Ahmad [40]	ResNet50	Facial Features	92%	Facial Images
Subah et al. [41]	DNN	Brain atlases	88%	rs-fMRI
Tang et al. [42]	Deep multimodal model	fMRI scan and ROI signal intensities	74%	fMRI
Han et al. [43]	MMSDAE	Feature fusion	95.56%	EEG and ET
Kong et al. [44]	DNN	Individual brain network with connectivity features between pairs of ROIs	90.39%	MRI from ABIDE I
Liu et al., 2020 [45]	DFC	MTFS	76.8%	fMRI from ABIDE I
Arya et al. [46]	3D CNN-GCN model	Feature fusion	64.23%	rs-fMRI
Eslami et al. [47]	ASD-DiagNet	Correlated and anticorrelated connections of the brain	70.3%	fMRI from ABIDE-I
Zhang et al. [48]	SC-CNN	Temporal feature	68.6%	Re-fMRI
Rahman and Subashini, [49]	MobileNet Xception EfficientNet B0 EfficientNet B1 EfficientNet B2	Static facial features	92.81%, 96.63%, 93.38%, 95.06%, 94.31%	Face photos
Wang et al. [50]	maLRR	AAL	74.62%	fMRI
Baygin et al. [51]	Hybrid Lightweight Deep Feature Generation (MobileNetV2, ShuffleNet, SqueezeNet)	Deep feature	96.44%	EEG
Zhang et al. [52]	GCN	Deepfusion	95%	EEG
Wang et al. [53]	DL with SVM-RFE	Feature self-taught learning network	93.59%	rs-fMRI
Haweel et al. [23]	CNN	Speech task facial features	80%	sMRI, TfMRI and rs-fMRI
Abbas et al. [54]	DeepMNF	Spatio temporal features	75%	rs-fMRI and sMRI
Rakhimberdina Z, Liu, and Murata [55]	Graph-based multi-model ensemble	RSFC and phenotypic features	73.13%	fMRI from ABIDE
Mostafa and Wu [56]	CAE	Lines, shapes, specific objects	96.2%	T1-weighted MRI, rs-fMRI
Sherkatghanad et al. [57]	CNN	Connectomes	70.22%	rs-fMRI from ABIDE

shows a summary of multimodal DL techniques using neuroimaging and non-neuroimaging techniques in ASD detection.

4. Modalities Used in ASD Detection

In the detection of autism spectrum disorder (ASD), a combination of neuroimaging and non-neuroimaging techniques is employed to assess various aspects of an individual’s behavior, cognition and neurological function.

Table 2. Various deep learning ASD detection modalities using neuroimaging and non-neuroimaging techniques and their description.

Neuroimaging
Functional Magnetic Resonance Imaging (fMRI)	FMRI measures brain blood flow, revealing activity and connectivity. FMRI aids in ASD detection using DL to analyze activation patterns and neural circuits.
Electroencephalography (EEG)	EEG captures brain signals. Enabling deep learning models to identify ASD-related patterns in brain activity through electrodes.
Electromyography (EMG)	EMG measures muscle electrical activity, revealing motor function and ASD impairments. Used in deep learning techniques to detect motor abnormalities early.
Non-Neuroimaging
Eye-Tracking (ET)	Eye-tracking technology monitors eye movements and gaze patterns Enabling deep learning models to identify ASD-related gaze behaviors, aiding social communication assessment.
Speech and Language Analysis	Identifying distinctive speech characteristics. Deep learning models analyze speech data to extract acoustic, prosodic, and linguistic features for diagnosing ASD,
Behavioral Data	It observes and assesses an individual’s behavior. Deep learning models identify ASD traits and patterns, improving accuracy and reliability through integration with other modalities.
Genetic Data	Genetic data in ASD detection enhances research, enhancing diagnosis and treatment strategies. DL integration with neuroimaging and behavioral data.

gives an overview of both types of modalities used to detect autism spectrum disorder at an early stage.

5. Deep Learning Models

Various neural network models are pivotal in improving ASD detection. CNNs excel at tasks like facial analysis, eye-tracking, and speech analysis, enhancing diagnostic accuracy and enabling personalized interventions [30,31,32]. DNNs are proficient at extracting complex patterns, aiding early detection, diagnosis, and personalized interventions [37,41,44]. RNNs are instrumental when analyzing sequential data and speech transcripts, supporting early screening and personalized interventions [24]. GCNs contribute by capturing relationships in neuroimaging and social interaction graphs, improving diagnostic accuracy and advancing ASD research [27,39,52]. These neural network models collectively enhance ASD detection across diverse data modalities.

6. Performance Analysis

The analysis shown in Table 1 presents the accuracy results from various ASD classification models, demonstrating advancements in deep learning for ASD detection. High-performing models include ASDL (98.2%), EfficientNetB1 (95.06%), CNN (94%), and MobileNet (95%). Competitively performing models include CNN (87.50%) and GCN (79.5%), while BLSTM (68.18%) and Graph-based multi-model ensemble (73.13%) achieve lower accuracies. This underscores the importance of choosing appropriate deep learning architectures for accurate ASD classification, offering insights for future research and clinical applications.

7. Research Gaps and Future Directions

The literature review identified several limitations in ASD detection, including limited multimodal data-based studies, lack of longitudinal studies, lack of explainability and interpretability, limited data size, and overall limitations. These issues require future research to address neuroimaging, genetic information, and behavioral assessments for improved accuracy and reliability. Addressing these issues is crucial for enhancing effectiveness and reliability in diverse datasets and populations.

Multimodal data integration in ASD detection faces challenges in feature integration, interpretability, and data consistency. Robust fusion techniques are needed for resource-intensive data collection. Collaboration with clinicians is crucial for practical effectiveness. Online learning and adaptive models are essential. Longitudinal analysis is crucial for personalized treatment plans. Innovations in DL models improve prediction accuracy and treatment strategies.

8. Conclusions

This review highlights recent advancements in early ASD detection using multimodal deep learning techniques, enhancing accuracy and objectivity. These techniques integrate behavioral, genetic, and neuroimaging data, enabling personalized interventions and standardized assessment processes. However, further research is needed to address challenges like improved detection accuracy, data availability and interpretability. Multimodal deep learning techniques for early ASD detection offer significant scientific implications, improving accuracy and reducing diagnostic inconsistencies. By integrating behavioral, genetic, and neuroimaging data, these techniques enable standardized assessments, personalized interventions, and large-scale screening. However, further research and validation are needed before widespread implementation in clinical settings.