Revealing Continuous Brain Dynamical Organization with Multimodal Graph Transformer

Chongyue Zhao, Liang Zhan, Paul M. Thompson, Heng Huang

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

4 Scopus citations

Abstract

Brain large-scale dynamics is constrained by the heterogeneity of intrinsic anatomical substrate. Little is known how the spatio-temporal dynamics adapt for the heterogeneous structural connectivity (SC). Modern neuroimaging modalities make it possible to study the intrinsic brain activity at the scale of seconds to minutes. Diffusion magnetic resonance imaging (dMRI) and functional MRI reveals the large-scale SC across different brain regions. Electrophysiological methods (i.e. MEG/EEG) provide direct measures of neural activity and exhibits complex neurobiological temporal dynamics which could not be solved by fMRI. However, most of existing multimodal analytical methods collapse the brain measurements either in space or time domain and fail to capture the spatio-temporal circuit dynamics. In this paper, we propose a novel spatio-temporal graph Transformer model to integrate the structural and functional connectivity in both spatial and temporal domain. The proposed method learns the heterogeneous node and graph representation via contrastive learning and multi-head attention based graph Transformer using multimodal brain data (i.e. fMRI, MRI, MEG and behavior performance). The proposed contrastive graph Transformer representation model incorporates the heterogeneity map constrained by T1-to-T2-weighted (T1w/T2w) to improve the model fit to structure-function interactions. The experimental results with multimodal resting state brain measurements demonstrate the proposed method could highlight the local properties of large-scale brain spatio-temporal dynamics and capture the dependence strength between functional connectivity and behaviors. In summary, the proposed method enables the complex brain dynamics explanation for different modal variants.

Original languageEnglish
Title of host publicationMedical Image Computing and Computer Assisted Intervention – MICCAI 2022 - 25th International Conference, Proceedings
EditorsLinwei Wang, Qi Dou, P. Thomas Fletcher, Stefanie Speidel, Shuo Li
PublisherSpringer Science and Business Media Deutschland GmbH
Pages346-355
Number of pages10
ISBN (Print)9783031164309
DOIs
StatePublished - 2022
Externally publishedYes
Event25th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2022 - Singapore, Singapore
Duration: 18 Sep 202222 Sep 2022

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13431 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference25th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2022
Country/TerritorySingapore
CitySingapore
Period18/09/2222/09/22

Keywords

  • Graph contrastive representation
  • Multimodal graph transformer
  • Neural graph differential equations

Fingerprint

Dive into the research topics of 'Revealing Continuous Brain Dynamical Organization with Multimodal Graph Transformer'. Together they form a unique fingerprint.

Cite this