IEEE Transactions on Multimedia最新文献_第2页

Hierarchy-Aware Multimodal Distillation for Recommendation 基于层次感知的多模式推荐蒸馏

IF 9.7 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-12 DOI: 10.1109/TMM.2026.3651049

Meng Jian;Tuo Wang;Meijuan Yang;Lifang Wu

{"title":"Hierarchy-Aware Multimodal Distillation for Recommendation","authors":"Meng Jian;Tuo Wang;Meijuan Yang;Lifang Wu","doi":"10.1109/TMM.2026.3651049","DOIUrl":"https://doi.org/10.1109/TMM.2026.3651049","url":null,"abstract":"Beyond behavioral interaction records, multimedia recommendation scenarios possess abundant semantic signals, which provide excellent data support for user interest mining. Recently, the multimodal enhanced interaction graph has been actively explored and has achieved great progress. However, these methods overlook the capability disparity of various modalities in learning users' interests and lack the ability to explore the hierarchical relationships of interests in modality, resulting in suboptimal recommendation performance. Therefore, this work investigates intra-modality hierarchical learning and inter-modality guidance, proposing a hyperbolic self-distillation (HSD) model for multimedia recommendation. In each modality space, HSD introduces a hyperbolic propagation to filter users' hierarchical interests from the interaction graph effectively. Inter-modality interests are aligned further by a two-level self-distillation strategy to designate multimodal interactions to teach single-modal learning, aiming at teaching and learning to promote each other. Extensive experiments on four public datasets demonstrate that the proposed HSD outperforms leading baselines for multimedia recommendation, verifying the effectiveness of hierarchical propagation and two-level self-distillation in mining users' hierarchical interests.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"2279-2290"},"PeriodicalIF":9.7,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147557759","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Multi-Modal Motion Retrieval by Learning a Fine-Grained Joint Embedding Space 学习细粒度关节嵌入空间的多模态运动检索

IF 9.7 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-15 DOI: 10.1109/TMM.2026.3654382

Shiyao Yu;Zi-An Wang;Kangning Yin;Zheng Tian;Mingyuan Zhang;Weixin Si;Shihao Zou

{"title":"Multi-Modal Motion Retrieval by Learning a Fine-Grained Joint Embedding Space","authors":"Shiyao Yu;Zi-An Wang;Kangning Yin;Zheng Tian;Mingyuan Zhang;Weixin Si;Shihao Zou","doi":"10.1109/TMM.2026.3654382","DOIUrl":"https://doi.org/10.1109/TMM.2026.3654382","url":null,"abstract":"Motion retrieval is crucial for motion acquisition, offering superior precision, realism, controllability, and editability compared to motion generation. Existing approaches leverage contrastive learning to construct a unified embedding space for motion retrieval from text or visual modality. However, these methods lack a more intuitive and user-friendly interaction mode and often overlook the sequential representation of most modalities for improved retrieval performance. To address these limitations, we propose a framework that aligns four modalities—text, audio, video, and motion—within a fine-grained joint embedding space, incorporating audio for the first time in motion retrieval to enhance user immersion and convenience. This fine-grained space is achieved through a sequence-level contrastive learning approach, which captures critical details across modalities for better alignment. To evaluate our framework, we augment existing text-motion datasets with synthetic but diverse audio recordings, creating two multi-modal motion retrieval datasets. Experimental results demonstrate superior performance over state-of-the-art methods across multiple sub-tasks, including an 10.16% improvement in R@10 for text-to-motion retrieval and a 25.43% improvement in R@1 for video-to-motion retrieval on the HumanML3D dataset. Furthermore, our results show that our 4-modal framework significantly outperforms its 3-modal counterpart, underscoring the potential of multi-modal motion retrieval for advancing motion acquisition.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"3694-3708"},"PeriodicalIF":9.7,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148003372","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Heterogeneous Multimodal Federated Learning With Missing Modality via Mask-Restoration and Self-Guidance 基于掩模恢复和自引导的异构多模态缺失联邦学习

IF 9.7 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-15 DOI: 10.1109/TMM.2026.3654379

Zhibo Cao;Kuangrong Hao;Lingguang Hao;Bing Wei;Lihong Ren

{"title":"Heterogeneous Multimodal Federated Learning With Missing Modality via Mask-Restoration and Self-Guidance","authors":"Zhibo Cao;Kuangrong Hao;Lingguang Hao;Bing Wei;Lihong Ren","doi":"10.1109/TMM.2026.3654379","DOIUrl":"https://doi.org/10.1109/TMM.2026.3654379","url":null,"abstract":"Federated learning (FL) is well-suited for multimodal tasks due to its ability to protect privacy and support local training. However, the complexity of real-world sensor environments causes modality heterogeneity across clients. Some modalities may be missing altogether, making it difficult to construct a generalized global model. Existing multimodal federated learning methods often address modality-missing scenarios under simplified assumptions of modality heterogeneity, typically focusing on unimodal clients and modality-complete multimodal clients. Moreover, to mitigate performance degradation caused by missing modalities, some approaches assume the availability of auxiliary information at the server, which may be impractical in real-world scenarios. Therefore, we propose a novel heterogeneous multimodal Federated Learning with Mask-Restoration and Self-Guidance (FL-MRSG). The Mask-Restoration employs a masking strategy to simulate missing data during feature extraction, enabling the network to learn semantic features of missing modality. Furthermore, we introduce an innovative self-guidance mechanism that leverages the restored data as guidance information, enabling the network to distinguish between complete and missing data representations. In addition, we propose a personalized decoupled aggregation strategy to facilitate the collaborative training of a global model across heterogeneous modality clients. We extend the multimodal test set to arbitrary modality combinations to evaluate the robustness of the global model. Extensive experiments on MOSI and SIMS datasets demonstrate the effectiveness of the proposed FL-MRSG for arbitrary missing modalities.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"3571-3583"},"PeriodicalIF":9.7,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148003466","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Learning Dual Modality Interactions for Event-Based Motion Deblurring 学习基于事件的运动去模糊的双模态交互

IF 9.7 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-16 DOI: 10.1109/TMM.2026.3654340

Zeyu Xiao;Zhuoyuan Li;Yang Zhao;Yu Liu;Zhao Zhang;Wei Jia

{"title":"Learning Dual Modality Interactions for Event-Based Motion Deblurring","authors":"Zeyu Xiao;Zhuoyuan Li;Yang Zhao;Yu Liu;Zhao Zhang;Wei Jia","doi":"10.1109/TMM.2026.3654340","DOIUrl":"https://doi.org/10.1109/TMM.2026.3654340","url":null,"abstract":"Event cameras hold great potential for motion deblurring because they capture motion information with microsecond precision, offering robustness to motion blur. However, the limited interaction between RGB frames and event streams presents a significant challenge, preventing the full utilization of the event cameras’ unique advantages. To address this, we propose <underline>Du</u>al frame-event <underline>Int</u>eraction and introduce a multi-scale <underline>Net</u>work structure, DuInt-Net. DuInt-Net aims to tackle two key challenges: (1) enhancing the representational and interaction capabilities between RGB frames and event streams, and (2) adaptively selecting richer visual features for improved motion deblurring. We introduce an event-frame joint interaction module that consists of three branches: a base branch, a global awareness attention branch, and a local enhancement attention branch. The base branch processes essential pixel-level features that retain the original structural information. The global branch integrates event data to improve large-scale motion understanding, while the local branch uses large-kernel convolutions to refine fine-grained details in RGB frames. For superior reconstruction performance, we also propose the event-guided multi-scale fusion attention module, which effectively combines local visual information and global frame-event relationships. Extensive experiments demonstrate that DuInt-Net achieves superior performance, both quantitatively and qualitatively, showcasing its superior motion deblurring capabilities.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"3652-3666"},"PeriodicalIF":9.7,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148003576","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

NIDC: General Task Backbone for Neuroimaging Analysis via Interpretable Deep Clustering 通过可解释的深度聚类进行神经成像分析的一般任务主干

IF 9.7 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-16 DOI: 10.1109/TMM.2026.3654428

Jiayu Ye;An Zeng;Dan Pan;Junhao Chen;Jingliang Zhao;Yiqun Zhang;Yang Liu

{"title":"NIDC: General Task Backbone for Neuroimaging Analysis via Interpretable Deep Clustering","authors":"Jiayu Ye;An Zeng;Dan Pan;Junhao Chen;Jingliang Zhao;Yiqun Zhang;Yang Liu","doi":"10.1109/TMM.2026.3654428","DOIUrl":"https://doi.org/10.1109/TMM.2026.3654428","url":null,"abstract":"Clustering techniques offer strong interpretability. However, they have significant limitations in the deep learning area due to their difficulty in capturing complex data structures, such as spatial and contextual information. This issue is especially pronounced in neuroimaging research, where high-dimensional and complex data greatly restricts the feature representation capability of clustering models. Hence, we propose a Neuroimaging Deep Clustering (NIDC) backbone network. We convert 3D neuroimagings into point sets and design clustering-based paradigms for context feature aggregation, feature interaction, and feature dispatching to enable deep feature extraction from the point sets. To better capture spatial information, we propose brain spatial relative position encoding, which assists the clustering paradigm in better understanding the anatomical structure of brain tissue and the positional relationships between different regions. Additionally, we design a sample center loss function to encourage tighter clustering of labels or voxels/feature points of the same class in the feature space, aiming to suppress both inter-class and intra-class similarities. Meanwhile, NIDC preserves the interpretability of traditional clustering techniques, allowing it to uncover relationships between brain regions and trace the decision-making process of each voxel. NIDC achieves highly competitive performance across multiple datasets in various downstream tasks, emerging as a new and practical solution for neuroimaging analysis.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"3958-3973"},"PeriodicalIF":9.7,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148171447","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

FoodDiff: A Collaborative Relationship Perception Framework for Food Image Synthesis Using Diffusion Models FoodDiff：使用扩散模型的食物图像合成的协作关系感知框架

IF 9.7 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-20 DOI: 10.1109/TMM.2026.3654435

Mengling Xu;Sisi You;Bing-Kun Bao

{"title":"FoodDiff: A Collaborative Relationship Perception Framework for Food Image Synthesis Using Diffusion Models","authors":"Mengling Xu;Sisi You;Bing-Kun Bao","doi":"10.1109/TMM.2026.3654435","DOIUrl":"https://doi.org/10.1109/TMM.2026.3654435","url":null,"abstract":"Food image generation is a typical application of text-to-image (T2I) models. The core difference between food image synthesis and other T2I tasks is that there exist complex collaborative relationships among ingredients, cooking actions, and food images, which determine the appearance of dishes. However, existing food image generation models generally ignore or fail to sufficiently utilize such collaborative relationships, which hinders the model from precisely perceiving the shapes and details of food. Furthermore, the pre-training distribution of T2I models is usually noisy and differs from the user-preferred food feature distributions, resulting in deviations from human aesthetics. To address the above issues, we propose <bold>FoodDiff</b>, a collaborative relationship-aware diffusion model for food image generation, which consists of three key components: (1) To perceive collaborative relationships, we propose a collaborative relation module to extract these relations and inject them into the image generation process. (2) To sufficiently interact with the relationships between recipe semantics and food representations, we propose a recipe fusion fine-tuning module to precisely fuse recipe semantics with visual features and fine-tune the pre-trained model. (3) To make the pre-training feature distribution conform to human preference, we introduce an image reward feedback mechanism to optimize the aesthetics of food images. In addition, we propose a high-quality food dataset named Food-Aesthetic with exquisite plates and elaborate annotations. Extensive experiments and human evaluations show that FoodDiff has superior image aesthetics and semantic consistency.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"3893-3904"},"PeriodicalIF":9.7,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148171909","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Multi-Constraint Relational Semantic Alignment Toward Image-Text Retrieval 面向图像-文本检索的多约束关系语义对齐

IF 9.9 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-02-23 DOI: 10.1109/TMM.2026.3664964

Yi Xiao;Jun Zhang;Aiguo Song

{"title":"Multi-Constraint Relational Semantic Alignment Toward Image-Text Retrieval","authors":"Yi Xiao;Jun Zhang;Aiguo Song","doi":"10.1109/TMM.2026.3664964","DOIUrl":"https://doi.org/10.1109/TMM.2026.3664964","url":null,"abstract":"Image-text retrieval aims to align image regions with textual words for semantic matching, facilitating bidirectional retrieval between images and texts. While significant progress has been made in modeling both coarse-grained image-sentence and fine-grained region-word relationships, fully capturing multi-granularity correspondences remains a challenge. Many existing methods predominantly depend on region-level segmentation or recognition, which tends to introduce noise, compromise semantic consistency, and increase computational complexity, ultimately limiting retrieval performance. To address these issues, we propose a Multi-constraint Relational Semantic Alignment (McRSA) method, which incorporates three complementary loss-based constraints to enhance multi-granularity alignment while preserving complete information. Specifically, the method includes Posterior Probability Estimation (PPE), which utilizes Bayesian analysis to model causal relationships between image-text feature pairs and labels, reducing intra-class variations for fine-grained alignment. Additionally, a Momentum-driven Centroid Update (MCU) mechanism is introduced to mitigate oscillations and improve modal consistency in coarse-grained representations. A dynamic Feature Scale Adaptation (FSA) module is also employed, adjusting feature scales across modalities to alleviate granularity discrepancies and improve alignment robustness. Extensive experiments on five public datasets (Flickr30K, MS-COCO, RSTPReid, CUHK-PEDES, and ICFG-PEDES) demonstrate that McRSA achieves competitive retrieval performance compared to existing methods.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"5106-5117"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148489039","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Multi-Granularity Superpoint Graph Learning for Weakly Supervised 3D Semantic Segmentation 弱监督三维语义分割的多粒度点图学习

IF 9.9 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-02-04 DOI: 10.1109/TMM.2026.3654421

Yan Fan;Yu Wang;Pengfei Zhu;Le Hui;Jin Xie;Bin Xiao;Qinghua Hu

{"title":"Multi-Granularity Superpoint Graph Learning for Weakly Supervised 3D Semantic Segmentation","authors":"Yan Fan;Yu Wang;Pengfei Zhu;Le Hui;Jin Xie;Bin Xiao;Qinghua Hu","doi":"10.1109/TMM.2026.3654421","DOIUrl":"https://doi.org/10.1109/TMM.2026.3654421","url":null,"abstract":"Weakly supervised 3D semantic segmentation has proven effective in alleviating the heavy dependence on dense annotations by generating high-quality pseudo-labels. However, due to the scene complexity and disorder of the point cloud, merely applying the model semantic prediction or hand-crafted feature similarity for pseudo labeling is inefficient and biased. This limitation inevitably results in incorrect pseudo labels. To tackle this challenge, we propose a new method called Multi-granularity Superpoint Graph Learning (MSGL) that leverages the multi-scale local features of point clouds to improve the quality of pseudo labels. We first design a multi-granularity local representation learning module on the superpoint graph to capture the neighboring structure information of each superpoint within complex scenes. Subsequently, the generated structural embedding is utilized to enhance the affinity matrix of label propagation, thereby yielding high-quality pseudo labels. To further enforce the generalization of the structural representation module under scenario changes or data fluctuations, we present a multi-granularity consistency loss in MSGL. This loss is applied across different views of the superpoint graph within each scene to ensure a robust and consistent learning process. Our experiments conducted on three benchmarks show that the proposed method outperforms existing weakly supervised methods under several sparse label settings, and improves the baseline by an average of 7.7% with only 1% extra computation cost. Moreover, our approach even compares favorably to some fully supervised methods with only one point labeled for each thing.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"4245-4258"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148489177","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Visibility-Based Geometry Pruning of Neural Plenoptic Scene Representations 基于可视性的神经全视场景表示几何剪枝

IF 9.9 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2025-10-06 DOI: 10.1109/TMM.2025.3618548

Davi R. Freitas;Ioan Tabus;Christine Guillemot

{"title":"Visibility-Based Geometry Pruning of Neural Plenoptic Scene Representations","authors":"Davi R. Freitas;Ioan Tabus;Christine Guillemot","doi":"10.1109/TMM.2025.3618548","DOIUrl":"https://doi.org/10.1109/TMM.2025.3618548","url":null,"abstract":"The need for more realistic 3D scene representations has fomented the development of models for a wide range of applications. In this context, solutions that attempt to model the light’s behavior through the plenoptic function have provided considerable advancements using neural-based approaches, often presenting a trade-off between rendering time and model sizes. In this work, we propose a pruning framework to reduce the sizes of these models by computing the visibility over the training data, applicable to different 3D scene representations. In particular, we implement first a solution suitable for the 3D Gaussian Splatting, and then we exemplify the solution for the Neural Radiance Fields (NeRF)-style of rendering using PlenOctrees. We show that our pruning solution produces smaller models in terms of the number of elements – be they voxels, points, or Gaussians – with minimal losses in terms of rendering novel views. We further assess our solution by combining it with state-of-the-art (SOTA) compression solutions for both rendering schemes. Results over the NeRF-Synthetic dataset show comparable metrics to the SOTA for PlenOctrees, achieving marginal gains for lower bitrates. For 3DGS, the combination of our pruning method and compression solutions achieves a compression ratio of up to 37.5 times over the uncompressed 3DGS models, with only a 0.5 dB decrease in rendering quality. When compared against other SOTA compression methods, our solution produces models 1.4 times smaller, with less than a 0.1 dB loss over novel views for synthetic data, and models 1.9 times smaller with less than 0.2 dB loss when synthesizing novel views on real-world, outdoor content.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"1-16"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148489181","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Question Understanding and Temporality Guiding for Video Question Answering 视频问答的问题理解与时间性引导

IF 9.9 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-13 DOI: 10.1109/TMM.2026.3651064

Jiahao Lu;Sisi You;Bing-Kun Bao

{"title":"Question Understanding and Temporality Guiding for Video Question Answering","authors":"Jiahao Lu;Sisi You;Bing-Kun Bao","doi":"10.1109/TMM.2026.3651064","DOIUrl":"https://doi.org/10.1109/TMM.2026.3651064","url":null,"abstract":"Video Question Answering (VideoQA) aims to answer a question based on the content of a given video. Recent methods adapt image-text pre-trained models to the VideoQA task by designing learnable temporal modules within the image encoder. However, these methods struggle to fully comprehend the questions and effectively extract temporal information due to 1) over-reliance on candidate answers and 2) lack of explicit temporal modeling. Specifically, since the question is fixed in different question-answer pairs, existing models tend to focus on the varying candidate answers. Moreover, existing methods merely utilize the classification loss to constrain the confidence of candidate answers, failing to differentiate the effectiveness of temporal information and to explicitly guide temporal modeling. In this paper, we introduce the Question Understanding and Temporality Guiding (QU-TG) method to address the aforementioned limitations. To reduce over-reliance on candidate answers, we propose providing diverse questions through question selection and enhancing the model’s comprehensive understanding of questions through question-video matching. To conduct explicit temporal modeling guiding, we propose negative video prevention and positive video guidance to conduct explicit temporal modeling guiding. Negative video prevention incorporates a prevention loss to discourage the model from making predictions based on erroneous temporal cues, whereas positive video guidance utilizes classification loss to encourage the model to derive correct answers from positive videos. Extensive experiments on the NExT-QA, IntentQA, STAR-QA, and Causal-VidQA datasets demonstrate the effectiveness and generalization of our method.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"2772-2783"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148489375","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0