IEEE Transactions on Multimedia最新文献

{"title":"Deep Learning-Driven Segmentation of Unseen Objects in Indoor Robotic Environments","authors":"Ying Zhang;Maoliang Yin;Qiao Zhang;Cui-Hua Zhang;Simon X. Yang;Zhijun Li;Changchun Hua","doi":"10.1109/TMM.2026.3673587","DOIUrl":"https://doi.org/10.1109/TMM.2026.3673587","url":null,"abstract":"For robots operating in unstructured indoor environments, the ability to accurately perceive and interact with invisible objects is crucial because it is infeasible and impractical to assume that every object in the environment is modeled. Despite advances in this direction, recognizing unseen objects remains a challenging perceptual task. The success of deep learning (DL) in many fields has promoted the development of DL-based methods to address this challenge. This paper proposes a comprehensive review of DL-driven Unseen Object Instance Segmentation (UOIS). To the best of our knowledge, this paper is the first to present the state-of-the-art solutions for UOIS, since existing reviews focused on either known object segmentation, on learning-based network models, or on some part of these, and little effort can be found on DL-based UOIS. In addition to conducting a comprehensive evaluation of existing literature on UOIS, we also classify DL-based UOIS methods according to their technical characteristics. After that, popular synthetic and benchmark datasets for UOIS are outlined, and a performance comparison of DL-based UOIS on the datasets is given. Finally, current challenges are discussed, and potential research opportunities are suggested.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"7607-7620"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149034146","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Privacy-Preserving Continuous Sign Language Recognition","authors":"Wanli Xue;Bitao Tan;Leming Guo;Tiantian Yuan;Yunpeng Li;Jia Guo;Shengyong Chen","doi":"10.1109/TMM.2026.3668639","DOIUrl":"https://doi.org/10.1109/TMM.2026.3668639","url":null,"abstract":"Responsible continuous sign language recognition (CSLR) needs to consider the privacy concerns of hearing-impaired people, which are ignored by CSLR methods. And simple privacy-preserving methods, such as face blurring, replacement, and pruning low-frequency components, degrade recognition performance. In this study, we propose PrivSign, the first privacy-preserving framework for CSLR. PrivSign transforms the RGB video sequence into the frequency domain and prunes low- to partial high-frequency components. The generated high-frequency videos are invisible to the human eye and conceal biometric information effectively. To solve the dilemma between concealing biological information and maintaining recognition performance, PrivSign conducts the Semantic Rectification. Specifically, the RGB-Frequency pre-training between RGB and high-frequency video enables the CSLR model to learn comprehensive sign semantics. Furthermore, the fine-tuning process combined with learnable Gabor filters is performed on high-frequency videos to mitigate noise and emphasize the significance of sign-relevant semantics. Recognition performance and black and gloss box attack results demonstrate that PrivSign achieves a balance among concealing biological information, impeding recovery, and recognition performance.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"6716-6727"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149034288","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Avatron: 3D Face Avatar and Voice Generation From Text","authors":"Seyun Um;Yongju Lee;WooSeok Ko;Yuan Zhou;Sangyoun Lee;Hong-Goo Kang","doi":"10.1109/TMM.2026.3660162","DOIUrl":"https://doi.org/10.1109/TMM.2026.3660162","url":null,"abstract":"Speech-driven 3D facial animation models are pivotal components in a variety of human interaction systems. However, speech-driven methods are strongly dependent on the quality of input speech, and there are no ways of flexibly changing the content of the speech without re-recording, which can be costly and time-consuming. Using text-to-speech (TTS) as an intermediate step offers an avenue to generate 3D avatars for any content, but involves an inefficient pipeline: converting text to speech and subsequently generating a 3D avatar from the synthesized speech. In order to overcome this constraint, we propose a novel approach for 3D facial animation that effectively merges these two domains. Our model takes advantage of intermediate features derived from the alignment of text and speech, which are then utilized as input for an avatar decoder to transform them into the vertices of a 3D facial animation. By jointly modeling lip movements, text, and speech characteristics, we are able to achieve exceptional performance while substantially reducing the number of model parameters and computational complexity. We demonstrate the efficacy of our model by assessing it in terms of the mean square error of 3D mesh vertices and through a user study that includes ABX test comparisons.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"4759-4770"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149034289","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Security and Real-Time FPGA Integration for Learned Image Compression","authors":"Alaa Eddine Mazouz;Carl De Sousa Trias;Sumanta Chaudhuri;Marco Cagnazzo;Mihai Mitrea;Enzo Tartaglione;Attilio Fiandrotti","doi":"10.1109/TMM.2026.3676828","DOIUrl":"https://doi.org/10.1109/TMM.2026.3676828","url":null,"abstract":"Learned Image Compression (LIC) is gaining traction nowadays, yet real-time performance and secure operations on hardware platforms remain challenging. This work addresses both challenges by presenting an integrated workflow for training, securing, and deploying LIC models on hardware. To achieve a hardware-efficient LIC model, we employ an iterative pruning and quantization process within a standard end-to-end learning framework. Additionally, we introduce <bold>Quantization-Aware Watermarking (QAW)</b>, a novel technique that embeds a watermark during quantization via a joint loss function, ensuring model integrity and security without degrading video performance. The watermarked weights undergo public-key encryption, enhancing protection by safeguarding both content and user traceability. We evaluate real-time performance, latency, energy consumption, and compression efficiency across two Field Programmable Gate Array (FPGA) platforms, showing that the watermarking and encryption steps introduce minimal overhead, PSNR decreases by 0.2 dB on average, energy consumption increases by 2%, and FPS drops by 6% on average while maintaining real-time constraints and security. Furthermore, our approach outperforms existing hardware-based LIC implementations in FPS and energy efficiency, delivering optimized LIC codecs for HD, FHD, and UHD resolutions at 61, 24, and 14 FPS, respectively.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"7747-7758"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149034365","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"SOPERM-Track: Spatial Object Permanence Modeling via Height Stability and Ground Perspective","authors":"Lyuchao Liao;Yukun Su;Zhaoxuan Lu;Feng Chen;Shaowei Weng","doi":"10.1109/TMM.2026.3731058","DOIUrl":"https://doi.org/10.1109/TMM.2026.3731058","url":null,"abstract":"Multi-object tracking is a crucial task in intelligent surveillance, yet it remains challenging in crowded environments due to severe occlusion, perspective distortion, and appearance similarity. To address these challenges, this work proposes <italic>Spatial Object Permanence Track</i> (SOPERM-Track), which aims to endow a motion-based tracker with human-like spatial object permanence by reconstructing spatiotemporal consistency through physical geometric constraints. Specifically, short-term height stability maintains scale continuity when object observations become unreliable due to occlusion, ground contact distance models ground-plane spatial relationships from the bottom edges of detection boxes, and a global perspective map learns scene-level perspective patterns online for global scale calibration. These complementary spatial cues form an appearance-free association mechanism to improve identity consistency under occlusion, interaction, and non-linear motion. On the MOT17, MOT20, and DanceTrack benchmarks and on the shuttlecock trajectory set, SOPERM-Track demonstrates competitive tracking performance and runtime efficiency in scenarios involving long-term occlusion, similar appearances, complex motion patterns, and fast tiny-object trajectory tracking.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"7813-7825"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149034379","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Language-Inspired Unknown-Space Expansion for Few-Shot Open-Set Detection","authors":"Zhimeng Xin;Tianxu Wu;Yixiong Zou;Shiming Chen;Qinmu Peng;Xinge You","doi":"10.1109/TMM.2026.3673530","DOIUrl":"https://doi.org/10.1109/TMM.2026.3673530","url":null,"abstract":"<bold>F</b>ew-shot <bold>o</b>pen-set <bold>o</b>bject <bold>d</b>etection (FOOD) aims to effectively detect known objects with a limited number of annotated samples while rejecting unknown ones. Due to the lack of real unknown data during training, existing methods treat high-uncertainty samples, e.g., background objects of known images, as simulated unknown objects for model optimization. However, such simulated unknown data can hardly reflect the distribution of real unknown data due to its similarity to the known data, and the limited number of known data even exacerbates this problem. To tackle this issue, we are inspired by large language models (LLM) for their rich prior knowledge to expand the simulated unknown data. We propose the Unknown Feature Space Expansion (UFSE) module to take simulated unknown data as a word that describes the rational unknown space for the given known data, and transforms them into inputs for a common text encoder, enriching the simulated unknown features. Following this, we develop an Unknown Pseudo-Classifier (UPC) as another component of UFSE to effectively identify the expanded unknown features from the background, further increasing the difference between unknown and known samples and improving models’ ability to distinguish real unknown data. Extensive experiments on the VOC and COCO datasets show that our proposed UFSE module achieves the state-of-the-art performance of FOOD, with 14.6% improvement in the mean average precision of novel classes in the VOC 1-shot setting.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"7697-7708"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149034364","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"TunCLIP: Adaptive Multi-Scale Zero-Shot Anomaly Detection via Vision-Language Feature Infiltration and Semantic Enhancement","authors":"Tao Zhang;Wenbo Pang;Xuequan Lu","doi":"10.1109/TMM.2026.3684998","DOIUrl":"https://doi.org/10.1109/TMM.2026.3684998","url":null,"abstract":"Zero-shot anomaly detection (ZSAD) is critical for identifying unseen anomalies without prior abnormal samples, particularly in safety-critical domains such as industrial inspection and medical diagnostics. While Vision-Language Models (VLMs) like CLIP show significant promise, existing CLIP-based ZSAD methods often struggle with adaptive anomaly region exploration, effective multi-scale feature fusion, and the extraction of subtle semantic anomalies. To address these limitations, this paper introduces TunCLIP, a novel and unified framework designed to enhance ZSAD performance through vision-language feature infiltration and semantic augmentation. TunCLIP integrates three complementary core modules: 1) Adaptive Anomaly-Probing Attentive Feature Infiltration (AAFI), which dynamically highlights and amplifies potential anomalous regions by generating spatial attention masks and injecting learnable tokens; 2) Hierarchical Multi-scale Latent Anomaly Spectrum Decoder (H-LASD), which captures cross-scale anomaly signals by modeling residual deviations against estimated normal baselines across multi-layer features; and 3) Anomaly Semantic Enhancement (ASE), which amplifies weak anomaly semantics by inverting channel affinity relationships to emphasize less cohesive, anomaly-indicative patterns. Extensive experiments on diverse industrial benchmarks (MVTec-AD, VisA, BTAD, DTD-Synthetic) and medical datasets (HeadCT, BrainMRI, ColonDB, TN3k, BMAD) demonstrate TunCLIP's superior performance, achieving average image-level AUROC of 90.2% and pixel-level AUROC of 95.8% on industrial datasets, and 95.6% and 87.3% respectively on medical datasets, consistently outperforming state-of-the-art methods. This research provides a robust, interpretable, and highly effective solution for ZSAD in complex real-world multimedia environments.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"7568-7578"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149034371","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"MF-GGNN: Crack Visual Reasoning CAPTCHA Holistically Using a Novel Multi-Feature Fusion-Based Graph Gated Neural Network","authors":"Botao Xu;Haizhou Wang;Wenxian Wang;Xingshu Chen","doi":"10.1109/TMM.2025.3645614","DOIUrl":"https://doi.org/10.1109/TMM.2025.3645614","url":null,"abstract":"With the advancement of computer vision and deep learning algorithms, many traditional text-based and image-based CAPTCHAs (Completely Automated Public Turing test to tell Computers and Humans Apart) have faced the significant risk of being automatically recognized and cracked. To enhance security, some leading CAPTCHA service providers have begun to explore interactive CAPTCHAs that integrate visual reasoning tasks, which greatly increase machine recognition difficulty. Compared to traditional CAPTCHAs, these CAPTCHAs leverage the unique advantage of the human brain in comprehensively analyzing and judging real-world information. Existing visual reasoning CAPTCHA research employs non-end-to-end model. However, it is unable to fully learn the rich features from CAPTCHAs through holistically modeling the logical relations. Additionally, the modular approach requires training modules separately and assembling them at the end, leading to substantial manual effort. To address the issues mentioned above, this paper presents an end-to-end learning framework called MF-GGNN (Multi-Feature Fusion-based Graph Gated Neural Network). The framework utilizes GGNN to simulate human association and reasoning over visual content. Specifically, we first locate objects in the image using an object detection model. We then extract multiple features and generate question attention distribution. Finally, we integrate the above outputs using a graph neural gated network for multi-step reasoning. Our model can capture the deep intrinsic connections between objects and their spatial information to achieve end-to-end graph reasoning. MF-GGNN achieves an average attack success rate of 92.2% across multiple CAPTCHAs. The results of various experiments demonstrate that our method can excellently accomplish visual reasoning tasks in CAPTCHAs. In addition, we construct a large-scale, open-source dataset, namely ViRC (<underline>Vi</u>sual <underline>R</u>easoning <underline>C</u>APTCHAs), including over 50,000 CAPTCHAs and corresponding questions. Our research findings will provide references and insights for evaluating and designing more secure and robust next-generation CAPTCHAs.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"1790-1805"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149034346","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"CHFusion: A Cross-Modality High-Resolution Representation Framework for Infrared and Visible Image Fusion","authors":"Biaojian Jin;Rencan Nie;Jinde Cao;Ying Zhang;Dongyang Li","doi":"10.1109/TMM.2023.3294814","DOIUrl":"10.1109/TMM.2023.3294814","url":null,"abstract":"In our study, we proposed a novel infrared and visible image fusion framework based on cross-modality transfer and high-resolution representation, termed as CHFusion. On the one hand, the high-resolution representation backbone is devised to receive multi-scale information and maintain high-resolution representation. More specifically, the proposed method involves the pyramid cross-modality feature transfer module to achieve information interaction with different modalities and resolutions. In particular, we introduce the gradient block to obtain texture information of the source image and then supplement it with high-resolution features. We utilize the adaptive channel attention block to compress high-resolution features and then guide image reconstruction. Moreover, a cross-modality high-resolution aggregation block is used to integrate multi-scale information. On the other hand, we propose a difference-aware algorithm, which can generate a pair of weights and then use the weights to construct the difference-aware loss, and then difference-aware loss, a texture loss, and an intensity loss to guide our network to preserve abundant texture information and optimally salient target. Extensive experiments demonstrate the superiority of our method over state-of-the-art alternatives in terms of object maintenance and texture preservation.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"8784-8797"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"63097064","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Circular Window-Based Cascade Transformer for Online Action Detection","authors":"Shuqiang Cao;Weixin Luo;Bairui Wang;Ran Song;Wei Zhang;Lin Ma","doi":"10.1109/TMM.2026.3676664","DOIUrl":"https://doi.org/10.1109/TMM.2026.3676664","url":null,"abstract":"Online Action Detection (OAD) aims to identify ongoing actions in untrimmed video streams, which is critical for real-world applications. However, effectively modeling long-term temporal dependencies while maintaining real-time inference efficiency remains challenging. To address this challenge, we propose a circular window-based cascade Transformer (CWC-Trans), featuring three key components: (i) a window-based hierarchical architecture that captures fine-grained temporal dynamics, and a circular window updating mechanism that enables efficient long-term history modeling; (ii) an auxiliary offline action segmentation (OAS) task that guides the hierarchical architecture to learn discriminative long-term action patterns while suppressing background interference; and (iii) a cascade prediction refinement module with adaptive kernels that progressively refines action predictions. By sharing the classifier and refinement modules between OAD and OAS tasks, CWC-Trans integrates multi-level long- and short-term information across feature, probabilistic, and refined probabilistic levels. Extensive experiments on THUMOS’14, TVSeries, and HDD demonstrate that CWC-Trans achieves state-of-the-art performance with high inference efficiency.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"8151-8163"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149130726","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}