IEEE Transactions on Multimedia最新文献_第9页

Reliable Multi-View Clustering With Graph Neural Network 基于图神经网络的可靠多视图聚类

IF 9.9 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-03-21 DOI: 10.1109/TMM.2026.3695792

Bao Zeng;Jielong Lu;Yanxin Xie;Shunxin Xiao;Jianmin Li;Chaohui Tang;Da-Han Wang

{"title":"Reliable Multi-View Clustering With Graph Neural Network","authors":"Bao Zeng;Jielong Lu;Yanxin Xie;Shunxin Xiao;Jianmin Li;Chaohui Tang;Da-Han Wang","doi":"10.1109/TMM.2026.3695792","DOIUrl":"https://doi.org/10.1109/TMM.2026.3695792","url":null,"abstract":"Deep multi-view clustering (MVC) typically leverages deep neural networks to extract consistency and complementarity information across multiple views. As a representative approach, graph neural networks (GNN) gather significant attention due to its ability to model both topology and feature simultaneously, making them highly attractive for researchers working on multi-view scenarios. However, existing GNN-based MVC algorithms often overlook the inherent heterogeneity and noise in multi-view data, which introduces uncertainty into the embedding learning process. Moreover, the message passing mechanism of GNN may further amplify this uncertainty, leading to unreliable embeddings and poor clustering performance. To address the issues, we propose a model called Reliable Multi-View Clustering with Graph Neural Network (RM-GNN). The proposed model first uses view-specific graph convolution encoders to project features of different views into unified semantic space. Then, we introduce an adaptive fusion method that effectively combines the representations obtained from each view. Furthermore, we propose a similarity-driven evidence learning module to effectively identify and distinguish unreliable samples in the embedding space. Finally, we introduce a unified transformation matrix to enforce global topological consistency. Extensive experimental results on nine benchmark datasets demonstrate that our algorithm generally outperforms state-of-the-art methods.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"9792-9803"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149131817","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Curation From Thousands: Composing the Best-Aligned Diffusion Models With Improved Model Retrieval and Fusion 成千上万的策展：用改进的模型检索和融合组成最佳对齐的扩散模型

IF 9.9 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-03-18 DOI: 10.1109/TMM.2026.3694503

Pu Cao;Rupeng Yan;Xuekun Zhao;Feng Zhou;Wenyi Zhao;Qing Song;Lu Yang

{"title":"Curation From Thousands: Composing the Best-Aligned Diffusion Models With Improved Model Retrieval and Fusion","authors":"Pu Cao;Rupeng Yan;Xuekun Zhao;Feng Zhou;Wenyi Zhao;Qing Song;Lu Yang","doi":"10.1109/TMM.2026.3694503","DOIUrl":"https://doi.org/10.1109/TMM.2026.3694503","url":null,"abstract":"In recent years, the community has witnessed a surge of open-sourced diffusion models tailored for various specialized tasks. As the number of available models continues to grow rapidly, selecting and merging relevant models to meet user needs has become a critical challenge in real-world applications. Although prior studies have explored this task, their model retrieval and fusion strategies typically rely solely on user-provided textual prompts and off-the-shelf large language models, leading to suboptimal performance. In this work, we extend the user-provided information by incorporating a reference image and propose a multimodal model curation framework, ChamGen, allowing model curation to better align with user intent. Then, we conduct an in-depth investigation into model retrieval and propose a lightweight modality fusion method based on contrastive learning, which encodes both textual and visual information into a shared embedding space, significantly boosting retrieval performance. Furthermore, benefiting from the introduction of the reference image, we estimate the optimal combination weights by minimizing the denoising loss of the fused model with respect to the reference image. To avoid the overhead of iterative optimization, we reformulate the parameter weight learning problem as a guidance weight assignment task, enabling us to estimate fusion weights in a single forward pass. Experiments show that our retrieval framework achieves 76.72% Recall@1 and 95.43% Recall@100 on a 10,000-scale diffusion model dataset. And the generation quality is significantly improved when combined with our fusion strategy.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"9728-9738"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149131863","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Multimodal Hierarchical Graph Contrastive Learning With Diffusion-Enhanced for Multimedia-Based Recommendation 基于扩散增强的多模态层次图对比学习的多媒体推荐

IF 9.9 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-06-10 DOI: 10.1109/TMM.2026.3700771

Xian Mo;Jun Pang;Haihua Zhao;Guanghua Ding;Rui Tang;Jintao Gao;Hao Liu

{"title":"Multimodal Hierarchical Graph Contrastive Learning With Diffusion-Enhanced for Multimedia-Based Recommendation","authors":"Xian Mo;Jun Pang;Haihua Zhao;Guanghua Ding;Rui Tang;Jintao Gao;Hao Liu","doi":"10.1109/TMM.2026.3700771","DOIUrl":"https://doi.org/10.1109/TMM.2026.3700771","url":null,"abstract":"Multimedia-based recommendation is an essential tool of web applications, which has been successfully applied to e-commerce and content-sharing applications encompassing a vast amount of web multimedia content. This paper introduces a <underline>multimodal <underline>hierarchical graph contrastive learning architecture with <underline>diffusion-enhanced (MHDiff) for multimedia-based recommendation. Specifically, our MHDiff first designs a hierarchical modality-aware diffusion model to construct two multimodal hierarchical contrastive views: one at the semantic level and the other at the content level. For the semantic-level contrastive view, we incorporate semantic entities from items across different modalities to guide the generation of a user-item graph. For the content-level contrastive view, we integrate various modality features to guide the generation of a user-item graph. Consequently, our architecture combines a hierarchical modality-aware graph diffusion model with a hierarchical modality-aware contrastive learning paradigm. This approach effectively eliminates noise from multimodal content and enhance modality-aware user representation learning. Ultimately, a modality-aware main view is generated, leveraging the two multimodal hierarchical contrastive views to enhance node semantic information with minimal noise. Comprehensive experiments on multiple multimedia datasets demonstrate the superiority of our MHDiff on various state-of-the-art multimedia-based recommendation methods.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"10435-10446"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149187991","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Robust Text-to-Image Person Re-Identification via Neighbor Consistency-Based Correspondence Estimation 基于邻居一致性对应估计的鲁棒文本-图像人物再识别

IF 9.9 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-06-05 DOI: 10.1109/TMM.2026.3700768

Jinjie Liu;Hao Wu;Xueping Wang;Min Liu;Yaonan Wang

{"title":"Robust Text-to-Image Person Re-Identification via Neighbor Consistency-Based Correspondence Estimation","authors":"Jinjie Liu;Hao Wu;Xueping Wang;Min Liu;Yaonan Wang","doi":"10.1109/TMM.2026.3700768","DOIUrl":"https://doi.org/10.1109/TMM.2026.3700768","url":null,"abstract":"Text-to-image person re-identification (TIReID) retrieves target individuals using textual queries, demanding precise alignment of sparse visual-textual cues. Real-world training data, however, suffers from noisy correspondence (NC), i.e., misaligned image-text pairs from annotation errors or ambiguities. While existing NC methods work for generic retrieval, they falter in fine-grained TIReID (identity-rich but pair-limited environments); crucially, their per-sample loss filtering mistakenly discards discriminative hard samples essential for subtle identity distinctions. To address these limitations, we propose Neighbor Consistency-based Correspondence Estimation (NCCE), a unified framework that integrates cross-modal context modeling and progressive noise suppression. Specifically, the Neighbor-Consistent Noise Identification (NCNI) strategy moves beyond per-sample analysis by inferring textual descriptions from the visual neighbors of an image and evaluating the consistency between the aggregated neighbor-inferred text and the original text via weighted similarity, thus avoiding the pitfalls of loss-based filtering. Moreover, Modality-Shared Projection Learning (MSPL) addresses fine-grained mismatches by projecting both modalities into a unified space, enabling adaptive alignment through mutual regularization. Extensive experiments on three public TIReID benchmarks validate the superiority of the proposed method. Notably, on the ICFG-PEDES dataset with a 50% noise ratio, our NCCE method achieves state-of-the-art (SOTA) performance, obtaining a Rank-1 accuracy of 66.15% (+2.48% improvement) and a mAP of 40.82% (+3.51% improvement). Consistent improvements of 1-3% in both metrics are observed on the other two datasets. Furthermore, our method demonstrates strong generalizability by achieving new SOTA results on the general image retrieval benchmark MS-COCO5K.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"10068-10078"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149188580","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Transferable Adversarial Attack on Person Re-Identification via Direction-Aware Attack 通过方向感知攻击对人再识别的可转移对抗攻击

IF 9.9 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-06-22 DOI: 10.1109/TMM.2026.3705904

Zeze Tao;Jinjia Peng;Huibing Wang

{"title":"Transferable Adversarial Attack on Person Re-Identification via Direction-Aware Attack","authors":"Zeze Tao;Jinjia Peng;Huibing Wang","doi":"10.1109/TMM.2026.3705904","DOIUrl":"https://doi.org/10.1109/TMM.2026.3705904","url":null,"abstract":"Person re-identification (Re-ID) system is facing severe threats from adversarial examples. Existing attack methods mislead the matching results of the Re-ID system by modifying feature distances to make adversarial examples located in feature regions far away from those of clean samples.However, we find that these methods suffer from the problem of Random Walk, where the direction adversarial examples move away from clean examples in the feature space is random, leading to limited transferability of attacks on the Re-ID system. To address this issue, we propose a novel <bold>Direction-<bold>Aware <bold>Attack (DAA) method, which can provide a clear and optimal directional guidance for adversarial examples. The core idea of DAA is to make adversarial examples move in the direction of features that are nearest neighbors and not of the initial identity, and this direction represents the most direct path to escape from the decision region of the original identity in the feature space. In addition, considering the potential feature space overlap between the nearest neighbor and the current identity, we propose a direction-aware optimization that steers the adversarial sample toward the nearest neighbor initially, then maximizes its distance from the original identity. Extensive experimental results show that the proposed DAA significantly outperforms the existing state-of-the-art attack methods.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"10962-10972"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149188711","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

MM-SAG: Multi-Modal Semantic-Aware Graph for Multi-Label Image Classification of Kitchen Waste 基于多模态语义感知图的厨余垃圾多标签图像分类

IF 9.9 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-06-12 DOI: 10.1109/TMM.2026.3703323

Hai Qin;Jintao Li;Qiaokang Liang;Wei Sun;Dan Zhang;Yaonan Wang

{"title":"MM-SAG: Multi-Modal Semantic-Aware Graph for Multi-Label Image Classification of Kitchen Waste","authors":"Hai Qin;Jintao Li;Qiaokang Liang;Wei Sun;Dan Zhang;Yaonan Wang","doi":"10.1109/TMM.2026.3703323","DOIUrl":"https://doi.org/10.1109/TMM.2026.3703323","url":null,"abstract":"Multi-label image classification aims to recognize multiple object labels within an image. In the field of intelligent waste sorting, efficient classification can enhance the accuracy of robotic sorting. However, most existing waste classification tasks are single-label, and there is limited research on multi-label classification of urban kitchen waste, especially in complex backgrounds and diverse categories. To address this, we propose a Multi-Modal Semantic-Aware Graph (MM-SAG) framework, which includes a semantic-aware module designed for instance-level label relationship mining. The captured semantic features are then processed through graph convolution to generate a label correlation matrix, enhancing the efficiency and effectiveness of label correlation mining. To improve the integration of visual and linguistic modalities, we design an improved multi-head attention mechanism module. This module re-encodes and aligns visual and textual features, further enhancing feature extraction capabilities. Experimental results show that our proposed method achieves a mean Average Precision (mAP) of 83.1% on the MLKW dataset, delivering state-of-the-art performance. The method’s strong generalization capability is also validated on public datasets VOC2007 and MS-COCO.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"10422-10434"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149188884","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Task-Aware Multimodal Large Language Models for Distributed Acoustic Sensing–Based Seismic Monitoring 分布式声传感地震监测的任务感知多模态大语言模型

IF 9.9 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-06-22 DOI: 10.1109/TMM.2026.3705143

Honglin Guo;Zimeng Zhao;Chao Li;Yuting Su;Huayong Yang;Weizhi Nie

{"title":"Task-Aware Multimodal Large Language Models for Distributed Acoustic Sensing–Based Seismic Monitoring","authors":"Honglin Guo;Zimeng Zhao;Chao Li;Yuting Su;Huayong Yang;Weizhi Nie","doi":"10.1109/TMM.2026.3705143","DOIUrl":"https://doi.org/10.1109/TMM.2026.3705143","url":null,"abstract":"Seismic monitoring detects earthquake activity to investigate fundamental geophysical processes, advance understanding of earthquake mechanisms, and support hazard mitigation. Distributed acoustic sensing (DAS) uses existing fiber-optic cables to provide dense, long-range, high-resolution measurements over wide areas at low cost, showing strong potential for seismic monitoring. However, manually labeled DAS data are scarce and large annotated datasets are hard to obtain, which limits supervised deep-learning methods. In this paper, we present DAS-MLLM, a training-free framework that leverages multimodal large language models (MLLMs) for seismic monitoring. DAS-MLLM comprises three components: (1) a task-aware Visual Generator that adaptively selects informative spectro-temporal visualizations of DAS signals; (2) a composite-similarity Retriever that uses time- and frequency-domain features to retrieve relevant exemplars for in-context learning; and (3) a feedback-guided Self-Corrector that integrates unselected visualizations to refine predictions. We evaluate DAS-MLLM on the Ridgecrest dataset and on the Xinfengjiang DAS dataset from Guangdong, China (558 events, with a high proportion of microearthquakes); the latter also includes seismometer recordings for some of the same events, enabling direct comparison with seismometer-based methods. Results show that our method improves most metrics in earthquake detection, magnitude estimation, and hypocenter localization tasks. DAS-MLLM reduces localization error by up to 28.8% and improves detection F1-score by 19.4% compared with state-of-the-art methods.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"10786-10797"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149188946","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

YACT-Net: Asymmetric YUV Color Transfer for Reference-Based Colorization 基于参考着色的非对称YUV色彩转移

IF 9.7 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-15 DOI: 10.1109/TMM.2026.3654440

Shenghong Feng;Junhong Lin;Shufan Pei;Ting Zhang;Tiesong Zhao

{"title":"YACT-Net: Asymmetric YUV Color Transfer for Reference-Based Colorization","authors":"Shenghong Feng;Junhong Lin;Shufan Pei;Ting Zhang;Tiesong Zhao","doi":"10.1109/TMM.2026.3654440","DOIUrl":"https://doi.org/10.1109/TMM.2026.3654440","url":null,"abstract":"Multi-camera systems have become extensively integrated into smartphones. One prominent implementation employs a color-monochrome dual-camera system, which synergistically combines data from both sensors to produce high-quality color images. The dual-camera system can capture asymmetric image pairs, consisting of relatively high-resolution monochrome images and low-resolution color images. Existing methods do not effectively address the issues of the inconsistency in spectral and spatial resolutions between color-monochrome image pairs in asymmetric color transfer. In this paper, we propose a novel framework based on YUV color space to address these issues. First, we conduct some analyses and experiments on several color spaces, demonstrating the effectiveness of asymmetric color transfer in YUV color space. Second, we design a YUV-color-space-based Asymmetric Color Transfer Network (YACT-Net) for reference-based colorization. In this network, we use a restoration module and a chrominance reconstruction module that utilize YUV information from image pairs to generate high-quality color images. Third, we create a Stereo Color Transfer (SCT) dataset, which includes both synthetic and real-world datasets, to evaluate the performance of YACT-Net. Experimental results demonstrate that YACT-Net achieves state-of-the-art performance in both quantitative and qualitative results.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"4158-4168"},"PeriodicalIF":9.7,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148171911","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Ranking-Based Self-Supervised Representation Learning for Skeleton-Based Action Recognition 基于排名的自监督表示学习用于骨架动作识别

IF 9.7 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-15 DOI: 10.1109/TMM.2026.3654466

Bizhu Wu;Junliang Chen;Jinheng Xie;Qiufu Li;Jianfeng Ren;Ruibin Bai;Rong Qu;Linlin Shen

{"title":"Ranking-Based Self-Supervised Representation Learning for Skeleton-Based Action Recognition","authors":"Bizhu Wu;Junliang Chen;Jinheng Xie;Qiufu Li;Jianfeng Ren;Ruibin Bai;Rong Qu;Linlin Shen","doi":"10.1109/TMM.2026.3654466","DOIUrl":"https://doi.org/10.1109/TMM.2026.3654466","url":null,"abstract":"Recently, researchers have achieved significant results in the skeleton-based action recognition. To better model the skeleton sequences, we drive the encoder to learn more discriminative representations in the self-supervised setting. We find that instead of clustering feature vectors to assign pseudo labels for samples as in DeepCluster, ranking them is a more reasonable, reliable, and efficient way to learn more effective feature representations. With this intuition, we propose a novel self-supervised learning framework, <bold>DeepRank. Specifically, we rank triplets of skeleton sequences with the ranking labels, obtained from the relative distances among them. Besides, to deeply mine complementary discriminative information that exists in different modalities of skeleton sequences, we further propose <bold>Multi-<bold>View <bold>DeepRank (<bold>MV-DeepRank) to enable encoders to comprehensively learn complementary features from multiple modalities. Extensive experimental results on the NTU RGB+D, NTU RGB+D 120, PKU-MMD I, and PKU-MMD II datasets under various evaluation settings demonstrate the generality, transferability, and superiority of our proposed self-supervised learning frameworks. Notably, our frameworks surpass the previous methods that employ the same backbone networks as ours by at least 1.8% (ST-GCN) and 2.1% (STTFormer) under the finetuning setting. Additionally, DeepRank gains a significant advantage on computational complexities, <inline-formula><tex-math>$O(1)$</tex-math></inline-formula>, over the contrastive learning-based methods, <inline-formula><tex-math>$O(text{batch size})$</tex-math></inline-formula>, and the clustering-based methods, <inline-formula><tex-math>$O(text{number of clusters})$</tex-math></inline-formula>.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"4181-4193"},"PeriodicalIF":9.7,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148171952","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

RVQ-NVC: Taming RVQ-VAE for High Fidelity Neural Vibrotactile Compression RVQ-NVC: Taming RVQ-VAE用于高保真神经振动触觉压缩

IF 9.7 1区计算机科学

IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-15 DOI: 10.1109/TMM.2026.3654467

Dazhong He;Qian Liu

{"title":"RVQ-NVC: Taming RVQ-VAE for High Fidelity Neural Vibrotactile Compression","authors":"Dazhong He;Qian Liu","doi":"10.1109/TMM.2026.3654467","DOIUrl":"https://doi.org/10.1109/TMM.2026.3654467","url":null,"abstract":"With the rise of the metaverse, haptic technologies have gained significant attention, striving to enhance human-machine interactions by incorporating a sense of touch alongside audio and visual experiences. Vibrotactile signals, which deliver haptic information through vibrations, are commonly used in commercial haptic interaction devices. The IEEE and MPEG have begun efforts to standardize the data compression of these signals. However, achieving efficient compression without sacrificing fidelity continues to pose a significant challenge. Inspired by the state-of-the-art audio compression techniques, we develop a novel residual vector quantization (RVQ)-based neural vibrotactile codec (RVQ-NVC) that capitalizes on the similarities between vibrotactile and audio signals in both time and frequency domains, strategically adapting RVQ to suit vibrotactile data. The RVQ-NVC scheme consists of a fully convolutional encoder-decoder network combined with an RVQ module, trained in an end-to-end manner. We employ perceptual and adversarial loss functions to ensure the high-quality reconstruction of vibrotactile signals while enabling variable bitrate coding through structured dropout techniques. Experimental results reveal that RVQ-NVC excels in signal reconstruction at low compression ratios, maintains high perceptual quality at reduced bitrates, and supports low-latency real-time processing, making it particularly well-suited for haptic interaction applications.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"4014-4028"},"PeriodicalIF":9.7,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148171499","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0