IEEE Transactions on Multimedia最新文献

筛选
英文 中文
GeoEdgeFormer: 3D Point Cloud Saliency Detection via Edge-Enhanced Graph-Transformer Network GeoEdgeFormer:基于边缘增强图变压器网络的三维点云显著性检测
IF 9.9 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-02-26 DOI: 10.1109/TMM.2026.3668301
Zihao Tian;Pengjie Wang;Xuan Qi;Pingping Zhang;Shengfeng He
{"title":"GeoEdgeFormer: 3D Point Cloud Saliency Detection via Edge-Enhanced Graph-Transformer Network","authors":"Zihao Tian;Pengjie Wang;Xuan Qi;Pingping Zhang;Shengfeng He","doi":"10.1109/TMM.2026.3668301","DOIUrl":"https://doi.org/10.1109/TMM.2026.3668301","url":null,"abstract":"The goal of point cloud salient object detection is to identify and segment the most prominent areas or objects within a 3D point cloud. Research on point cloud SOD is still in its early stages, and many existing methods fail to fully utilize the rich geometric information inherent in point clouds. To address this limitation, we introduce GeoEdgeFormer, an effective Edge-Enhanced Transformer Network, designed specifically for 3D salient object detection. GeoEdgeFormer employs an encoder-decoder architecture featuring two novel components: the Residual Edge Convolution (REC) and the Global Contextual Transformer (GCT). In the encoder, we propose the REC, which is designed to maintain permutation invariance while capturing local geometric information. This component not only improves the model's ability to process complex point cloud data but also enhances its efficiency, making it suitable for dynamic. In the decoder, we introduce the GCT to learn scene-level contextual representations. The GCT integrates global semantics and multi-level features from the encoder into a cohesive global scene context. By effectively combining features from local and global levels, the model achieves a more comprehensive understanding of the scene's semantics, thereby enhancing its generalization ability. Extensive experiments on the PCSOD saliency dataset demonstrate that our proposed GeoEdgeFormer achieves state-of-the-art performance.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"5437-5449"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148442701","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
3D-VMSS: Distributed Trust and Visually Meaningful Secret Sharing for 3D Mesh Models 3D- vmss: 3D网格模型的分布式信任和视觉上有意义的秘密共享
IF 9.9 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-02-27 DOI: 10.1109/TMM.2026.3668681
Kai Gao;Shuying Xu;Ching-Chun Chang;Chin-Chen Chang
{"title":"3D-VMSS: Distributed Trust and Visually Meaningful Secret Sharing for 3D Mesh Models","authors":"Kai Gao;Shuying Xu;Ching-Chun Chang;Chin-Chen Chang","doi":"10.1109/TMM.2026.3668681","DOIUrl":"https://doi.org/10.1109/TMM.2026.3668681","url":null,"abstract":"As industrial systems increasingly rely on 3D mesh models to bridge the physical and digital domains, ensuring their secure and efficient management has become critical. While thumbnail-preserving encryption (TPE) has successfully balanced security and usability for 2D images, extending this concept to 3D models remains largely unexplored. In this work, a novel distributed trust and visually meaningful secret sharing scheme for 3D mesh models (3D-VMSS) is proposed. The distributed trust mechanism splits the model data among multiple participants, ensuring that no single party possesses sufficient information for reconstruction. The original model can only be reconstructed through collaboration among a predefined threshold of authenticated participants. This approach fundamentally differs from traditional single-key encryption by eliminating single points of failure and enabling flexible access control. The scheme segments vertex coordinates into hierarchical components and applies polynomial secret sharing to ensure confidentiality, generating visually meaningful shares that preserve recognizable geometric features while concealing sensitive details. To ensure integrity and resist collusion attacks, dual authentication mechanisms are incorporated. Furthermore, progressive reconstruction enables different quality levels based on participant collaboration. Experimental results demonstrate the scheme's effectiveness in balancing security and practical usability for distributed 3D model management.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"5577-5591"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148442927","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Gain From Give Up: Intuitive Data Augmentation Framework for Image Retrieval 放弃获得:直观的图像检索数据增强框架
IF 9.9 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-15 DOI: 10.1109/TMM.2026.3654462
Yi Lu;Yurong Qian;Shu Li;Jiaying Chen;Guangqi Yang;Yuning Huang
{"title":"Gain From Give Up: Intuitive Data Augmentation Framework for Image Retrieval","authors":"Yi Lu;Yurong Qian;Shu Li;Jiaying Chen;Guangqi Yang;Yuning Huang","doi":"10.1109/TMM.2026.3654462","DOIUrl":"https://doi.org/10.1109/TMM.2026.3654462","url":null,"abstract":"Modern deep hashing methods rely on data augmentation to fully realize their potential in the face of large-scale retrieval galleries and over-parameterized visual models. However, this work observes that mainstream label-preserving augmentation methods are unreliable for information retrieval because they lead to incomplete alignment between data and labels. This misalignment impairs metric losses in distinguishing original/augmented data during same-class clustering, compromising nearest-neighbor search efficacy. To address these issues, we propose an innovative plug-and-play data augmentation framework tailored for retrieval tasks, based on the concept of <bold>G</b>aining robust features by randomly <bold>G</b>iving up parts of the image (<bold>GG</b>). Inspired by the ease with which visual changes induced by discard transformations can be estimated, we design two intuitive augmentation methods along with corresponding semantic shift estimators to measure the semantic changes introduced by each operation. Additionally, we optimize the metric loss based on the semantic retention scores, guiding the metric objective to properly allocate gradients for generated samples. This adjustment mitigates the adverse effects caused by incomplete alignment, optimizing the intra-class distance of both original and augmented data in the Hamming space, while ensuring the relevance and accuracy of the retrieval results. Extensive experiments conducted on six datasets demonstrate the effectiveness and robustness of our proposed framework.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"4346-4359"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148443099","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Enhancing Cross-Domain Correspondence for Unsupervised Image-to-Image Translation 增强无监督图像到图像翻译的跨域对应
IF 9.9 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-02-02 DOI: 10.1109/TMM.2026.3660079
Binxin Lai;Wenkang Su;Yuying Liang;Yuan-Gen Wang;Mingjie Li;Jiantao Zhou
{"title":"Enhancing Cross-Domain Correspondence for Unsupervised Image-to-Image Translation","authors":"Binxin Lai;Wenkang Su;Yuying Liang;Yuan-Gen Wang;Mingjie Li;Jiantao Zhou","doi":"10.1109/TMM.2026.3660079","DOIUrl":"https://doi.org/10.1109/TMM.2026.3660079","url":null,"abstract":"UNsupervised Image-to-image Translation (UNIT) aims to translate images across visual domains without paired training data, which has been widely used in style transfer, image processing, game design, etc. However, ensuring the correspondence (e.g., target category, pose, or head orientation) between generated images and inputs remains a formidable challenge. To this end, we present a new scheme, named EC-UNIT, which comprises three innovative designs aiming to Enhance cross-domain Correspondence for UNIT. Specifically, 1) we propose Multi-level Style Embedding to extract multi-level style features for fusion while imposing our newly designed Hierarchical Consistency Constraints on both the content and style features (MSE&HCC), aiming to retain more style representations and facilitate feature disentanglement; 2) we develop Semantic Perceptual Matching (SPM) to minimize the semantic distribution discrepancy between the generated image and the input image by leveraging the multimodal model CLIP, dedicated to enhancing semantic consistency; 3) considering that previous works have struggled to control the image translation using pixel-level visual consistency constraints, we design Visual Perceptual Guidance (VPG) to reduce the perceptual distance between the generated image and the style input in VGG feature space, devoted to enhancing visual perceptual correspondence, thereby preventing the generation of unrealistic image details. Extensive experiments demonstrate that our EC-UNIT is more stable and outperforms current SOTA competitors in terms of image quality and diversity as well as both content and style consistency.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"4509-4524"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148443106","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
A Transformer-Based Tracker Integrating Motion and Representation Information 一种融合运动与表征信息的变压器跟踪器
IF 9.9 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-02-13 DOI: 10.1109/TMM.2026.3660077
Yuanhui Wang;Ben Ye;Zhanchuan Cai;Hao Wu
{"title":"A Transformer-Based Tracker Integrating Motion and Representation Information","authors":"Yuanhui Wang;Ben Ye;Zhanchuan Cai;Hao Wu","doi":"10.1109/TMM.2026.3660077","DOIUrl":"https://doi.org/10.1109/TMM.2026.3660077","url":null,"abstract":"The appearance information of the target has been used as the only tracking cue for most trackers to locate the target in the video. However, when the surrounding environment changes drastically or there are similar interference targets, it usually causes target drift. We propose a tracker named MRTrack, a transformer-based spatiotemporal information decoupling network architecture to enhance the target tracking capability. We designed two training schemes to explore the effective integration of motion cues derived from optical flow with representation information. The first approach integrates the target’s motion and representation information for training. The second scheme is a step-by-step training, where the target motion information is first learned, and the learned model is used for representation learning. We compare the two training methods on five generic tracking datasets. The experiment results indicate that the first training approach can better integrate motion and representation information, leading to more precise tracking results for MRTrack compared to solely relying on the appearance model. In addition, optical flow cues are used only in the training phase to guide the tracker in understanding motion information, and no additional cost is incurred during tracking inference.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"4383-4392"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148443117","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models MoE-LLaVA:大型视觉语言模型的混合专家
IF 9.9 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-15 DOI: 10.1109/TMM.2026.3654458
Bin Lin;Zhenyu Tang;Yang Ye;Jinfa Huang;Junwu Zhang;Yatian Pang;Peng Jin;Munan Ning;Jiebo Luo;Li Yuan
{"title":"MoE-LLaVA: Mixture of Experts for Large Vision-Language Models","authors":"Bin Lin;Zhenyu Tang;Yang Ye;Jinfa Huang;Junwu Zhang;Yatian Pang;Peng Jin;Munan Ning;Jiebo Luo;Li Yuan","doi":"10.1109/TMM.2026.3654458","DOIUrl":"https://doi.org/10.1109/TMM.2026.3654458","url":null,"abstract":"Recently, remarkable progress has been made in scaling up Large Language Models (LLMs) through the use of the sparse Mixture-of-Expert (MoE) layers without significantly increasing computational cost. However, the transition from a pre-trained LLM to a sparse Large Vision-Language Model (LVLM) with MoE remains an open challenge. Directly fine-tuning an LLM to a sparse LVLM often leads to training collapse, characterized by (1) a large modality feature distribution gap and (2) expert load imbalance. This paper proposes a three-stage decoupled weight training process. In the first two stages, the model learns to adapt the LLM to an LVLM. In the third stage, the FFN weights from the second stage are used as lossless initialization for expert weights, effectively constructing a sparse model with a vast number of parameters while maintaining constant computational cost. Through extensive ablation experiments, we derive three empirical guidelines and propose a sparse LVLM termed <bold>MoE-LLaVA</b>. MoE-LLaVA is a MoE-based sparse LVLM architecture, which uniquely activates only the top-<inline-formula><tex-math>$k$</tex-math></inline-formula> experts through routers during deployment, keeping the remaining experts inactive. Extensive experiments demonstrate that MoE-LLaVA outperforms LLaVA-1.5-7B by 4.6 on average across nine visual understanding benchmarks. Notably, MoE-LLaVA has only 2.2B active parameters, achieving an average performance of 66.7, surpassing LLaVA-1.5-7B’s 65.5. Our work establishes a baseline for sparse LVLMs and provides empirical guidelines for exploring the sparse LVLMs.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"4408-4419"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148443131","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
When Multi-Focus Image Fusion Meets Nonlinear Spiking Neural P Systems 多焦点图像融合与非线性脉冲神经系统的研究
IF 9.9 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2025-10-21 DOI: 10.1109/TMM.2025.3623493
Bo Li;Lingling Zhang;Tingting Bao;Yunkuo Lei;Xiaoqing Zhang;Jun Liu
{"title":"When Multi-Focus Image Fusion Meets Nonlinear Spiking Neural P Systems","authors":"Bo Li;Lingling Zhang;Tingting Bao;Yunkuo Lei;Xiaoqing Zhang;Jun Liu","doi":"10.1109/TMM.2025.3623493","DOIUrl":"https://doi.org/10.1109/TMM.2025.3623493","url":null,"abstract":"Multi-focus image fusion (MFIF) aims to combine multiple images captured in the same scene by imaging devices with varying focal lengths into one complete clear image. While artificial neural network-based methods have achieved remarkable results in MFIF tasks, their black-box working mechanisms easily leadto information losses, limiting further fusion performance improvements. To solve this issue, we introduce an interpretable neural computation model called the nonlinear spiking neural P (NSNP) system. The NSNP model effectively mitigates the information losses induced by neurons during the information transmission process by controlling the internal spike values of neurons. According to the NSNP model, we propose a novel fusion method, NSNPFuse, that effectively avoids information losses during the challenging MFIF task. On the one hand, NSNPFuse uses nonlinear spiking neurons to construct the network backbone, which yields improved feature extraction performance and reduces the induced feature loss. On the other hand, NSNPFuse embeds a feature fusion module (FFM) based on nonlinear spiking neurons to selectively retain meaningful information and reduce distortions. We conduct experiments on multiple multi-focus image datasets, including Lytro, MFFW, MFI-WHU, and Road-MF, and the subjective and objective performances of the proposed approach surpass those of 15 state-of-the-art MFIF methods. The results demonstrate that our NSNPFuse method offers more competitive performance. Furthermore, we show that NSNPFuse enhances the downstream performance achieved in salient detection and object detection tasks.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"545-560"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148487405","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
A Cross-Modal Shared Attention Fusion Framework for Emotion Recognition in Multi-Party Conversations 一种用于多人对话情感识别的跨模态共享注意融合框架
IF 9.9 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2025-12-03 DOI: 10.1109/TMM.2025.3639948
Shifei Ding;Jie Li;Lili Guo
{"title":"A Cross-Modal Shared Attention Fusion Framework for Emotion Recognition in Multi-Party Conversations","authors":"Shifei Ding;Jie Li;Lili Guo","doi":"10.1109/TMM.2025.3639948","DOIUrl":"https://doi.org/10.1109/TMM.2025.3639948","url":null,"abstract":"Emotion Recognition in Conversation (ERC) enables machines to respond emotionally to users’ requests, which is crucial in human-computer interaction. Existing methods concentrate on capturing context and speaker-dependent relationships within the text modality or emphasize multi-modal interaction and feature fusion. However, these methods fail to effectively align shared emotional representations across modalities and preserve unique emotional features within each modality. To address this challenge, we propose an end-to-end framework called Cross-Modal Shared Attention (ShareERC). Specifically, we first extract context-aware features from text and audio, aggregating emotional information within each modality. Leveraging this aggregated emotional information, we design a Cross-Modal Shared Attention module that aligns shared features across modalities in the feature space while preserving the distinct features of each modality. Extensive experiments on the MELD and IEMOCAP datasets demonstrate the effectiveness and superiority of the proposed ShareERC.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"1654-1662"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149033963","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Deep Understanding of Sign Language for Sign to Subtitle Alignment 手语对字幕对齐的深刻理解
IF 9.9 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-03-12 DOI: 10.1109/TMM.2026.3673520
Youngjoon Jang;Jeongsoo Choi;Junseok Ahn;Joon Son Chung
{"title":"Deep Understanding of Sign Language for Sign to Subtitle Alignment","authors":"Youngjoon Jang;Jeongsoo Choi;Junseok Ahn;Joon Son Chung","doi":"10.1109/TMM.2026.3673520","DOIUrl":"https://doi.org/10.1109/TMM.2026.3673520","url":null,"abstract":"The objective of this work is to align asynchronous subtitles in sign language videos with limited labelled data. To achieve this goal, we propose a novel framework with the following contributions: (1) we leverage fundamental grammatical rules of British Sign Language (BSL) to pre-process the input subtitles, (2) we design a selective alignment (SA) loss to optimise the model for predicting the temporal location of signs only when the queried sign actually occurs in a scene, and (3) we conduct self-training with refined pseudo-labels which are more accurate than the heuristic audio-aligned labels. From this, our model not only better understands the correlation between the text and the signs, but also holds potential for application in the translation of sign languages, particularly in scenarios where manual labelling of large-scale sign data is impractical or challenging. Extensive experimental results demonstrate that our approach achieves state-of-the-art results, surpassing previous baselines by substantial margins in terms of both frame-level accuracy and F1-score. This highlights the effectiveness and practicality of our framework in advancing the field of sign language video alignment and translation.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"7647-7659"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149033935","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
StarVid: Enhancing Semantic Alignment in Video Diffusion Models via Spatial and SynTactic Guided Attention Refocusing 通过空间和句法引导的注意力再聚焦增强视频扩散模型中的语义对齐
IF 9.9 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-03-12 DOI: 10.1109/TMM.2026.3668668
Yuanhang Li;Qi Mao;Lan Chen;Zhen Fang;Lei Tian;Xinyan Xiao;Libiao Jin;Hua Wu
{"title":"StarVid: Enhancing Semantic Alignment in Video Diffusion Models via Spatial and SynTactic Guided Attention Refocusing","authors":"Yuanhang Li;Qi Mao;Lan Chen;Zhen Fang;Lei Tian;Xinyan Xiao;Libiao Jin;Hua Wu","doi":"10.1109/TMM.2026.3668668","DOIUrl":"https://doi.org/10.1109/TMM.2026.3668668","url":null,"abstract":"Recent advances in text-to-video (T2V) generation with diffusion models have garnered significant attention. However, they typically perform well in scenes with a single object and motion, struggling in compositional scenarios with multiple objects and distinct motions to accurately reflect the semantic content of text prompts. To address these challenges, we propose <bold>StarVid</b>, a plug-and-play, training-free method that improves semantic alignment between multiple subjects, their motions, and text prompts in T2V models. StarVid first employs large language models (LLMs) to perform two-stage motion trajectory planning based on the input prompt, providing object-level spatial priors that guide the generation process. These priors are then used to impose a spatial-aware loss that refines cross-attention (CA) maps, encouraging attention to be focused on distinct regions corresponding to different subjects. Furthermore, a syntax-guided contrastive constraint is introduced to explicitly enhance the association between verbs and their corresponding nouns in the CA maps, thereby improving motion-subject binding. Both qualitative and quantitative evaluations demonstrate that the proposed framework significantly outperforms baseline methods, delivering videos of higher quality with improved semantic consistency.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"6688-6703"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149034070","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
相关产品
×
本文献相关产品
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信
小红书