ArXiv.org最新文献

筛选
英文 中文
I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts I2MoE:可解释的多模态交互感知混合专家
ArXiv.org Pub Date : 2025-05-25 DOI: 10.48550/arxiv.2505.19190
Jiayi Xin, Yun, Sukwon, Jie Peng, Inyoung Choi, Jenna L. Ballard, Tianlong Chen, Qi Long
{"title":"I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts","authors":"Jiayi Xin, Yun, Sukwon, Jie Peng, Inyoung Choi, Jenna L. Ballard, Tianlong Chen, Qi Long","doi":"10.48550/arxiv.2505.19190","DOIUrl":"https://doi.org/10.48550/arxiv.2505.19190","url":null,"abstract":"Modality fusion is a cornerstone of multimodal learning, enabling information integration from diverse data sources. However, vanilla fusion methods are limited by (1) inability to account for heterogeneous interactions between modalities and (2) lack of interpretability in uncovering the multimodal interactions inherent in the data. To this end, we propose I2MoE (Interpretable Multimodal Interaction-aware Mixture of Experts), an end-to-end MoE framework designed to enhance modality fusion by explicitly modeling diverse multimodal interactions, as well as providing interpretation on a local and global level. First, I2MoE utilizes different interaction experts with weakly supervised interaction losses to learn multimodal interactions in a data-driven way. Second, I2MoE deploys a reweighting model that assigns importance scores for the output of each interaction expert, which offers sample-level and dataset-level interpretation. Extensive evaluation of medical and general multimodal datasets shows that I2MoE is flexible enough to be combined with different fusion techniques, consistently improves task performance, and provides interpretation across various real-world scenarios. Code is available at https://github.com/Raina-Xin/I2MoE.","PeriodicalId":522188,"journal":{"name":"ArXiv.org","volume":"1 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-05-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://arxiv.org/pdf/2505.19190","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147893153","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
相关产品
×
本文献相关产品
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信
小红书