Information Fusion最新文献_第3页

Tokenized EEG signals with large language models for epilepsy detection via multimodal information fusion 基于多模态信息融合的大语言模型脑电信号标记化检测

IF 15.5 1区计算机科学

Information Fusion Pub Date : 2026-07-01 Epub Date: 2026-01-07 DOI: 10.1016/j.inffus.2026.104128

XingchiChen , Fushen Xie , Fa Zhu , Shuanglong Zhang , Xiaoyang Lu , Qing Li , Rong Chen , Dazhou Li , David Camacho

{"title":"Tokenized EEG signals with large language models for epilepsy detection via multimodal information fusion","authors":"XingchiChen , Fushen Xie , Fa Zhu , Shuanglong Zhang , Xiaoyang Lu , Qing Li , Rong Chen , Dazhou Li , David Camacho","doi":"10.1016/j.inffus.2026.104128","DOIUrl":"10.1016/j.inffus.2026.104128","url":null,"abstract":"<div><div>The detection of epileptic seizures using multi-sensor EEG signals is a challenging task due to the inherent complexity of the signals, the variability in sensor configurations, and the difficulty in distinguishing the weak inter-class difference. To address these challenges, we propose a novel multimodal information fusion framework that integrates a large language model (LLM) and a multimodal EEG feature tokenization method for enhanced epilepsy detection. This paper adopts a multimodal feature extraction (MFE) method to effectively generate multimodal feature representations from EEG signals and extract different feature representations of EEG signals from different signal domains. In addition, we design a multimodal EEG feature tokenization method to tokenize EEG signal features and fuse the semantic information, solving the problem of fusing epileptic EEG features with semantic information in prompt words. We use the powerful reasoning and pattern recognition capabilities of pre-trained LLMs to accurately and robustly detect epileptic events. The proposed method is evaluated on a public dataset. Extensive experimental results show that the proposed method outperforms the current comparative methods in multiple performance indicators.</div></div>","PeriodicalId":50367,"journal":{"name":"Information Fusion","volume":"131 ","pages":"Article 104128"},"PeriodicalIF":15.5,"publicationDate":"2026-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"146006527","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

StegaFusion: Steganography for information hiding and fusion in multimodality 隐写融合：多模态信息隐藏与融合的隐写技术

IF 15.5 1区计算机科学

Information Fusion Pub Date : 2026-07-01 Epub Date: 2026-01-22 DOI: 10.1016/j.inffus.2026.104150

Zihao Xu , Dawei Xu , Zihan Li , Juan Hu , Baokun Zheng , Chuan Zhang , Liehuang Zhu

{"title":"StegaFusion: Steganography for information hiding and fusion in multimodality","authors":"Zihao Xu , Dawei Xu , Zihan Li , Juan Hu , Baokun Zheng , Chuan Zhang , Liehuang Zhu","doi":"10.1016/j.inffus.2026.104150","DOIUrl":"10.1016/j.inffus.2026.104150","url":null,"abstract":"<div><div>Current generative steganography techniques have attracted considerable attention due to their security. However, different platforms and social environments exhibit varying preferred modalities, and existing generative steganography techniques are often restricted to a single modality. Inspired by advancements in inpainting techniques, we observe that the inpainting process is inherently generative. Moreover, cross-modal inpainting minimally perturbs unchanged regions and shares a consistent masking-and-fill procedure. Based on these insights, we introduce StegaFusion, a novel framework for unifying multimodal generative steganography. StegaFusion leverages shared generation seeds and conditional information, which enables the receiver to deterministically reconstruct the reference content. The receiver then performs differential analysis on the inpainting-generated stego content to extract the secret message. Compared to traditional unimodal methods, StegaFusion enhances controllability, security, compatibility, and interpretability without requiring additional model training. To the best of our knowledge, StegaFusion is the first framework to formalize and unify cross-modal generative steganography, offering wide applicability. Extensive qualitative and quantitative experiments demonstrate the superior performance of StegaFusion in terms of controllability, security, and cross-modal compatibility.</div></div>","PeriodicalId":50367,"journal":{"name":"Information Fusion","volume":"131 ","pages":"Article 104150"},"PeriodicalIF":15.5,"publicationDate":"2026-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"146033292","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

EDGCN: An embedding-driven fusion framework for heterogeneity-aware motor imagery decoding 基于嵌入驱动的异构感知运动图像解码融合框架

IF 15.5 1区计算机科学

Information Fusion Pub Date : 2026-07-01 Epub Date: 2026-01-22 DOI: 10.1016/j.inffus.2026.104170

Chaowen Shen, Yanwen Zhang, Zejing Zhao, Akio Namiki

{"title":"EDGCN: An embedding-driven fusion framework for heterogeneity-aware motor imagery decoding","authors":"Chaowen Shen, Yanwen Zhang, Zejing Zhao, Akio Namiki","doi":"10.1016/j.inffus.2026.104170","DOIUrl":"10.1016/j.inffus.2026.104170","url":null,"abstract":"<div><div>Motor imagery electroencephalography (MI-EEG) captures neural activity associated with imagined motor tasks and has been widely applied in both basic neuroscience and clinical research. However, the intrinsic spatio-temporal heterogeneity of MI-EEG signals and pronounced inter-subject variability present major challenges for accurate decoding. Most existing deep learning methods rely on fixed architectures and shared parameters, which limits their ability to capture the complex, dynamic patterns driven by individual differences. To address these limitations, we propose an Embedding-Driven Graph Convolutional Network (EDGCN), which leverages a heterogeneity-aware spatio-temporal embedding fusion mechanism to adaptively generate graph convolutional kernel parameters from a shared embedding-driven parameter bank. Specifically, we design a Multi-Resolution Temporal Embedding (MRTE) strategy based on multi-resolution power spectral features and a Structure-Aware Spatial Embedding (SASE) mechanism that integrates both local and global connectivity structures. On this basis, we construct a heterogeneity-aware parameter generation mechanism based on Chebyshev graph convolution to effectively capture the spatiotemporal heterogeneity of EEG signals, with an orthogonality-constrained parameter space that enhances diversity and representational fusion. Experimental results demonstrate that the proposed model achieves superior classification accuracies of 86.50% and 90.14% on the BCIC-IV-2a and BCIC-IV-2b datasets, respectively, outperforming current state-of-the-art methods.</div></div>","PeriodicalId":50367,"journal":{"name":"Information Fusion","volume":"131 ","pages":"Article 104170"},"PeriodicalIF":15.5,"publicationDate":"2026-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"146033916","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Multimodal spatio-temporal fusion: A generalizable GCN-LSTM with attention framework for urban application 多模态时空融合：一个具有城市应用关注框架的广义GCN-LSTM

IF 15.5 1区计算机科学

Information Fusion Pub Date : 2026-07-01 Epub Date: 2026-01-20 DOI: 10.1016/j.inffus.2026.104164

Yunfei Guo

{"title":"Multimodal spatio-temporal fusion: A generalizable GCN-LSTM with attention framework for urban application","authors":"Yunfei Guo","doi":"10.1016/j.inffus.2026.104164","DOIUrl":"10.1016/j.inffus.2026.104164","url":null,"abstract":"<div><div>The proliferation of urban big data presents unprecedented opportunities for understanding cities, yet the analytical methods to harness this data are often fragmented and domain-specific. Existing predictive models in urban computing are typically highly specialized, creating analytical silos that inhibit knowledge transfer and are difficult to adapt across domains such as public safety, housing and transport. This paper confronts this critical gap by developing a generalizable, multimodal spatio-temporal deep learning framework engineered for both high predictive performance and interpretability, which is capable of mastering diverse urban prediction tasks without architectural modification. The hybrid architecture fuses a Multi-Head Graph Convolutional Network (GCN) for spatial diffusion, a Long Short-Term Memory (LSTM) network for temporal dynamics, and a learnable Gating Mechanism that weights the influence of spatial graph versus static external features. To validate this generalizability, the framework was tested on three distinct urban domains in London: crime forecasting, housing price estimation and transport network demand. The model outperformed traditional baselines (ARIMA, XGBoost) and state-of-the-art deep learning models (TabNet, TFT). Moreover, the framework moves beyond prediction to explanation by incorporating attention mechanisms and permutation feature importance analysis.</div></div>","PeriodicalId":50367,"journal":{"name":"Information Fusion","volume":"131 ","pages":"Article 104164"},"PeriodicalIF":15.5,"publicationDate":"2026-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"146014809","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Information-theoretic graph fusion with vision-language-action model for policy reasoning and dual robotic control 基于视觉语言-动作模型的信息图融合策略推理与双机器人控制

IF 15.5 1区计算机科学

Information Fusion Pub Date : 2026-07-01 Epub Date: 2026-01-29 DOI: 10.1016/j.inffus.2026.104193

Shunlei Li , Longsen Gao , Jin Wang , Chang Che , Xi Xiao , Jiuwen Cao , Yingbai Hu , Hamid Reza Karimi

{"title":"Information-theoretic graph fusion with vision-language-action model for policy reasoning and dual robotic control","authors":"Shunlei Li , Longsen Gao , Jin Wang , Chang Che , Xi Xiao , Jiuwen Cao , Yingbai Hu , Hamid Reza Karimi","doi":"10.1016/j.inffus.2026.104193","DOIUrl":"10.1016/j.inffus.2026.104193","url":null,"abstract":"<div><div>Teaching robots dexterous skills from human videos remains challenging due to the reliance on low-level trajectory imitation, which fails to generalize across object types, spatial layouts, and manipulator configurations. We propose Graph-Fused Vision-Language-Action (GF-VLA), a framework that enables dual-arm robotic systems to perform task-level reasoning and execution directly from RGB(-D) human demonstrations. GF-VLA first extracts Shannon-information-based cues to identify hands and objects with the highest task relevance, then encodes these cues into temporally ordered scene graphs that capture both hand-object and object-object interactions. These graphs are fused with a language-conditioned transformer that generates hierarchical behavior trees and interpretable Cartesian motion commands. To improve execution efficiency in bimanual settings, we further introduce a cross-hand selection policy that infers optimal gripper assignment without explicit geometric reasoning. We evaluate GF-VLA on four structured dual-arm block assembly tasks involving symbolic shape construction and spatial generalization. Experimental results show that the information-theoretic scene representation achieves over 95% graph accuracy and 93% subtask segmentation, supporting the LLM planner in generating reliable and human-readable task policies. When executed by the dual-arm robot, these policies yield 94% grasp success, 89% placement accuracy, and 90% overall task success across stacking, letter-building, and geometric reconfiguration scenarios, demonstrating strong generalization and robustness across diverse spatial and semantic variations.</div></div>","PeriodicalId":50367,"journal":{"name":"Information Fusion","volume":"131 ","pages":"Article 104193"},"PeriodicalIF":15.5,"publicationDate":"2026-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"146072488","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Vision-language model with siamese bilateral difference network and text-guided image feature enhancement for acute ischemic stroke outcome prediction on CT angiography 基于Siamese双侧差异网络和文本引导图像特征增强的视觉语言模型在急性缺血性卒中CT血管造影预后预测中的应用

IF 15.5 1区计算机科学

Information Fusion Pub Date : 2026-07-01 Epub Date: 2026-01-29 DOI: 10.1016/j.inffus.2026.104195

Hulin Kuang , Bin Hu , Shuai Yang , Dongcui Wang , Guanghua Luo , Weihua Liao , Wu Qiu , Shulin Liu , Jianxin Wang

{"title":"Vision-language model with siamese bilateral difference network and text-guided image feature enhancement for acute ischemic stroke outcome prediction on CT angiography","authors":"Hulin Kuang , Bin Hu , Shuai Yang , Dongcui Wang , Guanghua Luo , Weihua Liao , Wu Qiu , Shulin Liu , Jianxin Wang","doi":"10.1016/j.inffus.2026.104195","DOIUrl":"10.1016/j.inffus.2026.104195","url":null,"abstract":"<div><div>Acute ischemic stroke (AIS) outcome prediction is crucial for treatment decisions. However, AIS outcome prediction is challenging due to the combined influence of lesion characteristics, vascular status, and other health conditions. In this study, we introduce a vision-language model with a Siamese bilateral difference network and a text-guided image feature enhancement module for predicting AIS outcome (e.g., modified Rankin Scale, mRS) on CT angiography. In the Siamese bilateral difference network, based on fine-tuning the foundation model LVM-Med, we design an interactive Transformer fine-tuning encoder and a vision question answering guided bilateral difference awareness module, which generates bilateral difference text via image-text pair question answering as a prompt to enhance the extracted brain vascular difference features. Additionally, in the text-guided image feature enhancement module, we propose a text feature extraction module to extract patient phrase-level and inter-phrase embeddings from clinical notes, and employ a multi-scale image-text interaction module to obtain fine-grained phrase-enhanced image attention feature and coarse-grained phrase context-aware image attention feature. We validate our model on the public ISLES2024 dataset, a private dataset A, and an external AIS dataset. It achieves accuracies of 81.11%, 83.05%, and 80.00% and AUCs of 80.06%, 85.48% and 82.62% for 90-day mRS prediction on the 3 datasets, respectively, outperforming several state-of-the-art methods and demonstrating its generalization ability. Moreover, the proposed method can be effectively extended to glaucoma visual field progression prediction, which is also related to vascular differences and clinical notes.</div></div>","PeriodicalId":50367,"journal":{"name":"Information Fusion","volume":"131 ","pages":"Article 104195"},"PeriodicalIF":15.5,"publicationDate":"2026-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"146072487","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

On the security and privacy of federated learning: A survey with attacks, defenses, frameworks, applications, and future directions 关于联邦学习的安全和隐私：攻击、防御、框架、应用和未来方向的调查

IF 15.5 1区计算机科学

Information Fusion Pub Date : 2026-07-01 Epub Date: 2026-01-16 DOI: 10.1016/j.inffus.2026.104155

Daniel M. Jimenez-Gutierrez , Yelizaveta Falkouskaya , José L. Hernandez-Ramos , Aris Anagnostopoulos , Ioannis Chatzigiannakis , Andrea Vitaletti

{"title":"On the security and privacy of federated learning: A survey with attacks, defenses, frameworks, applications, and future directions","authors":"Daniel M. Jimenez-Gutierrez , Yelizaveta Falkouskaya , José L. Hernandez-Ramos , Aris Anagnostopoulos , Ioannis Chatzigiannakis , Andrea Vitaletti","doi":"10.1016/j.inffus.2026.104155","DOIUrl":"10.1016/j.inffus.2026.104155","url":null,"abstract":"<div><div>Federated Learning (FL) is an emerging distributed machine learning paradigm enabling multiple clients to train a global model collaboratively without sharing their raw data. While FL enhances data privacy by design, it remains vulnerable to various security and privacy threats. This survey provides a comprehensive overview of 203 papers regarding the state-of-the-art attacks and defense mechanisms developed to address these challenges, categorizing them into security-enhancing and privacy-preserving techniques. Security-enhancing methods aim to improve FL robustness against malicious behaviors such as byzantine attacks, poisoning, and Sybil attacks. At the same time, privacy-preserving techniques focus on protecting sensitive data through cryptographic approaches, differential privacy, and secure aggregation. We critically analyze the strengths and limitations of existing methods, highlight the trade-offs between privacy, security, and model performance, and discuss the implications of non-IID data distributions on the effectiveness of these defenses. Furthermore, we identify open research challenges and future directions, including the need for scalable, adaptive, and energy-efficient solutions operating in dynamic and heterogeneous FL environments. Our survey aims to guide researchers and practitioners in developing robust and privacy-preserving FL systems, fostering advancements safeguarding collaborative learning frameworks’ integrity and confidentiality.</div></div>","PeriodicalId":50367,"journal":{"name":"Information Fusion","volume":"131 ","pages":"Article 104155"},"PeriodicalIF":15.5,"publicationDate":"2026-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145995205","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Data fusion for low-cost sensors: A systematic literature review 低成本传感器的数据融合：系统文献综述

IF 15.5 1区计算机科学

Information Fusion Pub Date : 2026-07-01 Epub Date: 2026-01-18 DOI: 10.1016/j.inffus.2026.104124

Gabriel Oduori , Chaira Cocco , Payam Sajadi , Francesco Pilla

{"title":"Data fusion for low-cost sensors: A systematic literature review","authors":"Gabriel Oduori , Chaira Cocco , Payam Sajadi , Francesco Pilla","doi":"10.1016/j.inffus.2026.104124","DOIUrl":"10.1016/j.inffus.2026.104124","url":null,"abstract":"<div><div>Data fusion (DF) addresses the challenge of integrating heterogeneous data sources to improve decision-making and inference. Although DF has been widely explored, no prior systematic review has specifically focused on its application to low-cost sensor (LCS) data in environmental monitoring. To address this gap, we conduct a systematic literature review (SLR) following the PRISMA framework, synthesising findings from 82 peer-reviewed articles. The review addresses three key questions: (1) What fusion methodologies are employed in conjunction with LCS data? (2) In what environmental contexts are these methods applied? (3) What are the methodological challenges and research gaps? Our analysis reveals that geostatistical and machine learning approaches dominate current practice, with air quality monitoring emerging as the primary application domain. Additionally, artificial intelligence (AI)-based methods are increasingly used to integrate spatial, temporal, and multimodal data. However, limitations persist in uncertainty quantification, validation standards, and the generalisability of fusion frameworks. This review provides a comprehensive synthesis of current techniques and outlines key directions for future research, including the development of robust, uncertainty-aware fusion methods and broader application to less-studied environmental variables.</div></div>","PeriodicalId":50367,"journal":{"name":"Information Fusion","volume":"131 ","pages":"Article 104124"},"PeriodicalIF":15.5,"publicationDate":"2026-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145995206","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Grading-inspired complementary enhancing for multimodal sentiment analysis 基于评分的多模态情感分析互补增强

IF 15.5 1区计算机科学

Information Fusion Pub Date : 2026-07-01 Epub Date: 2026-01-23 DOI: 10.1016/j.inffus.2026.104174

Zhijing Huang , Wen-Jue He , Baotian Hu, Zheng Zhang

{"title":"Grading-inspired complementary enhancing for multimodal sentiment analysis","authors":"Zhijing Huang , Wen-Jue He , Baotian Hu, Zheng Zhang","doi":"10.1016/j.inffus.2026.104174","DOIUrl":"10.1016/j.inffus.2026.104174","url":null,"abstract":"<div><div>Due to its strong capacity for integrating heterogeneous multi-source information, multimodal sentiment analysis (MSA) has achieved remarkable progress in affective computing. However, existing methods typically adopt symmetric fusion strategies that treat all modalities equally, overlooking their inherent performance disparities that some modalities excel at discriminative representation, while others carry underutilized supportive cues. This limitation leads to insufficiency in cross-modal complementary correlation exploration. To address this issue, we propose a novel Grading-Inspired Complementary Enhancing (GCE) framework for MSA, which is one of the first attempts to conduct dynamic assessment for knowledge transfer in progressive multimodal fusion and cooperation. Specifically, based on cross-modal interaction, a task-aware grading mechanism categorizes modality-pair associations into dominant (high-performing) and supplementary (low-performing) branches according to their task performance. Accordingly, a relation filtering module selectively identifies the trustworthy information from the dominant branch to enhance consistency exploration in supplementary modality pairs with minimized redundancy. Afterwards, a weight adaptation module is adopted to dynamically adjust the guiding weight of individual samples for adaptability and generalization. Extensive experiments conducted on three benchmark datasets evidence that our proposed GCE approach can outperform the state-of-the-art MSA methods. Our code is available at <span><span>https://github.com/hka-7/GCEforMSA</span><svg><path></path></svg></span>.</div></div>","PeriodicalId":50367,"journal":{"name":"Information Fusion","volume":"131 ","pages":"Article 104174"},"PeriodicalIF":15.5,"publicationDate":"2026-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"146047985","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Shape-aware osteoarthritis network: Bidirectional fusion of MRI and 3D point clouds for knee osteoarthritis diagnosis 形状感知骨关节炎网络：MRI和3D点云双向融合诊断膝关节骨关节炎

IF 15.5 1区计算机科学

Information Fusion Pub Date : 2026-07-01 Epub Date: 2026-01-29 DOI: 10.1016/j.inffus.2026.104198

Dawei Zhang , Chenglin Sang , Tianyi Lyu

{"title":"Shape-aware osteoarthritis network: Bidirectional fusion of MRI and 3D point clouds for knee osteoarthritis diagnosis","authors":"Dawei Zhang , Chenglin Sang , Tianyi Lyu","doi":"10.1016/j.inffus.2026.104198","DOIUrl":"10.1016/j.inffus.2026.104198","url":null,"abstract":"<div><div>Knee osteoarthritis (KOA) is a common degenerative joint disease, and accurate diagnosis and severity grading are crucial for effective treatment. At present, although deep learning techniques based on X-rays or magnetic resonance imaging (MRI) have greatly improved diagnostic accuracy, two-dimensional images often cannot fully capture the complex three-dimensional morphology and texture changes related to KOA. To address these challenges, we propose a shape aware osteoarthritis diagnostic network, which is a novel bidirectional cross modal fusion framework that integrates 3D point clouds and MRI sequences. This framework consists of three parts: (1) a local relation aware dynamic graph convolutional neural network (CNN) used to extract complex geometric features from point clouds representing the surfaces of knee joint bones and cartilage; (2) For MRI sequences, a sequence aggregation method was adopted, which combines 2D CNN for spatial feature extraction and self-attention mechanism for cross slice sequences. (3) The bidirectional transmembrane fusion module is capable of conducting in-depth interactive feature learning between the geometric domain of point clouds and the texture spatiotemporal domain of MRI, enabling these two modes to improve and enhance each other’s representations. Extensive experiments conducted on a large cohort of osteoarthritis initiatives (OAI) have shown that our model achieves state-of-the-art performance. Its accuracy in the challenging 5-level Kellgren Lawrence (KL) classification is 0.73, which represents a improvement of approximately 23.7% over the 0.59 achieved by using 3D shape features alone in the ShapeMed-Knee benchmark. Furthermore, its AUC in binary OA diagnosis is 0.95, significantly better than existing unimodal and multimodal baselines.</div></div>","PeriodicalId":50367,"journal":{"name":"Information Fusion","volume":"131 ","pages":"Article 104198"},"PeriodicalIF":15.5,"publicationDate":"2026-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"146072486","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0