This is an early access version, the complete PDF, HTML, and XML versions will be available soon.
Open AccessArticle
Efficient Neural Decoding Based on Multimodal Training
by
Yun Wang
Yun Wang 1,2
1
Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai 200433, China
2
Key Laboratory of Computational Neuroscience and Brain-Inspired Intelligence, Fudan University, Ministry of Education, Shanghai 200433, China
Brain Sci. 2024, 14(10), 988; https://doi.org/10.3390/brainsci14100988 (registering DOI)
Submission received: 23 August 2024
/
Revised: 24 September 2024
/
Accepted: 26 September 2024
/
Published: 28 September 2024
Abstract
Background/Objectives: Neural decoding methods are often limited by the performance of brain encoders, which map complex brain signals into a latent representation space of perception information. These brain encoders are constrained by the limited amount of paired brain and stimuli data available for training, making it challenging to learn rich neural representations. Methods: To address this limitation, we present a novel multimodal training approach using paired image and functional magnetic resonance imaging (fMRI) data to establish a brain masked autoencoder that learns the interactions between images and brain activities. Subsequently, we employ a diffusion model conditioned on brain data to decode realistic images. Results: Our method achieves high-quality decoding results in semantic contents and low-level visual attributes, outperforming previous methods both qualitatively and quantitatively, while maintaining computational efficiency. Additionally, our method is applied to decode artificial patterns across region of interests (ROIs) to explore their functional properties. We not only validate existing knowledge concerning ROIs but also unveil new insights, such as the synergy between early visual cortex and higher-level scene ROIs, as well as the competition within the higher-level scene ROIs. Conclusions: These findings provide valuable insights for future directions in the field of neural decoding.
Share and Cite
MDPI and ACS Style
Wang, Y.
Efficient Neural Decoding Based on Multimodal Training. Brain Sci. 2024, 14, 988.
https://doi.org/10.3390/brainsci14100988
AMA Style
Wang Y.
Efficient Neural Decoding Based on Multimodal Training. Brain Sciences. 2024; 14(10):988.
https://doi.org/10.3390/brainsci14100988
Chicago/Turabian Style
Wang, Yun.
2024. "Efficient Neural Decoding Based on Multimodal Training" Brain Sciences 14, no. 10: 988.
https://doi.org/10.3390/brainsci14100988
Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details
here.
Article Metrics
Article metric data becomes available approximately 24 hours after publication online.