Multitask and Multimodal Neural Tuning for Large Models

Hao Sun, Yu Song, Jihong Hu, Yen-Wei Chen, Lanfen Lin
{"title":"Multitask and Multimodal Neural Tuning for Large Models","authors":"Hao Sun, Yu Song, Jihong Hu, Yen-Wei Chen, Lanfen Lin","doi":"arxiv-2408.03001","DOIUrl":null,"url":null,"abstract":"In recent years, large-scale multimodal models have demonstrated impressive\ncapabilities across various domains. However, enabling these models to\neffectively perform multiple multimodal tasks simultaneously remains a\nsignificant challenge. To address this, we introduce a novel tuning method\ncalled neural tuning, designed to handle diverse multimodal tasks concurrently,\nincluding reasoning segmentation, referring segmentation, image captioning, and\ntext-to-image generation. Neural tuning emulates sparse distributed\nrepresentation in human brain, where only specific subsets of neurons are\nactivated for each task. Additionally, we present a new benchmark, MMUD, where\neach sample is annotated with multiple task labels. By applying neural tuning\nto pretrained large models on the MMUD benchmark, we achieve simultaneous task\nhandling in a streamlined and efficient manner. All models, code, and datasets\nwill be publicly available after publication, facilitating further research and\ndevelopment in this field.","PeriodicalId":501480,"journal":{"name":"arXiv - CS - Multimedia","volume":"23 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-08-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Multimedia","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2408.03001","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

In recent years, large-scale multimodal models have demonstrated impressive capabilities across various domains. However, enabling these models to effectively perform multiple multimodal tasks simultaneously remains a significant challenge. To address this, we introduce a novel tuning method called neural tuning, designed to handle diverse multimodal tasks concurrently, including reasoning segmentation, referring segmentation, image captioning, and text-to-image generation. Neural tuning emulates sparse distributed representation in human brain, where only specific subsets of neurons are activated for each task. Additionally, we present a new benchmark, MMUD, where each sample is annotated with multiple task labels. By applying neural tuning to pretrained large models on the MMUD benchmark, we achieve simultaneous task handling in a streamlined and efficient manner. All models, code, and datasets will be publicly available after publication, facilitating further research and development in this field.
大型模型的多任务和多模态神经调整
近年来,大规模多模态模型在各个领域都表现出令人印象深刻的能力。然而,让这些模型同时有效地执行多种多模态任务仍然是一个重大挑战。为了解决这个问题,我们引入了一种称为神经调谐的新型调谐方法,旨在同时处理多种多模态任务,包括推理分割、指代分割、图像字幕和文本到图像生成。神经调谐模拟了人脑中的稀疏分布式表示,即每个任务只激活特定的神经元子集。此外,我们还提出了一个新的基准--MMUD,其中每个样本都标注了多个任务标签。通过在 MMUD 基准上对预训练的大型模型进行神经调谐,我们以精简高效的方式实现了同步任务处理。所有模型、代码和数据集都将在发表后公开,以促进该领域的进一步研究和开发。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信