IEEE Transactions on Multimedia最新文献

{"title":"FreeQR: Free Lunch for Aesthetic QR Codes Emerging From the Latent Space in Diffusion Models","authors":"Yiwei Yang;Jun Jia;Zheyuan Liu;Zhongpai Gao;Yunhao Li;Wei Sun;Guangtao Zhai","doi":"10.1109/TMM.2026.3668595","DOIUrl":"https://doi.org/10.1109/TMM.2026.3668595","url":null,"abstract":"In the modern digital age, Quick Response (QR) codes serve as a critical interface for bridging the physical and virtual worlds, widely utilized in multimedia applications. However, traditional binary QR codes often lack the visual appeal desired in contexts. Aesthetic QR codes address this limitation by enabling the customization of QR code patterns to enhance visual attractiveness while retaining compatibility with standard QR decoders. Previous works have explored the use of diffusion models for generating such codes but often require extensive training of ControlNets and face challenges in maintaining scannability. To address these issues, we present FreeQR, a streamlined and effective approach that enables the stable generation of QR code images with diffusion models. Our methodology involves the strategic fusion between the specific channel in the latent space of the denoising process with the noised latent representations of the QR blueprint image at corresponding timesteps. This ensures that the generated images adhere to the brightness distribution required for effective scanning while achieving a balance between aesthetics and functionality. Additionally, we introduce gradient guidance based on scanning errors directly in the latent space, enabling the generation of scannable QR codes in seconds without additional model parameters. Experimental results demonstrate that FreeQR significantly enhances the aesthetics and scannability of QR codes compared to existing methods, making it a lightweight and efficient solution for multimedia applications.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"6932-6943"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148747036","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"DeDiff-4DGS: Fusing Temporal Correlations and Diffusion Priors for Dynamic 3D Scenes","authors":"Hoang Nguyen Nguyen;Wei Xiang;Kang Han;Lu Yu;Phu Lai;Tianyu Chen;Yi-Ping Phoebe Chen","doi":"10.1109/TMM.2026.3673518","DOIUrl":"https://doi.org/10.1109/TMM.2026.3673518","url":null,"abstract":"Reconstructing dynamic 3D (4D) scenes is challenging due to complex temporal dynamics and viewpoint sparsity in monocular videos. Existing extensions of 3D Gaussian Splatting (3D-GS) with its temporal modeling often fail to capture temporal correlations across frames, leading to redundant 3D Gaussians and reduced efficiency. To address this limitation, we propose DeDiff-4DGS, a framework that integrates temporal correlations and diffusion priors through two novel modules. The Temporal 3D Gaussian Latent Fusion (T3DLF) module fuses temporal information from sparse reference frames to promote spatio-temporal coherence and reduce the number of required 3D Gaussians. The Latent Diffusion Converter for 3D Gaussians (LDC3D) module enriches reference frames with semantic priors, complementing T3DLF under sparse-view conditions. Experimental results on standard benchmarks demonstrate that DeDiff-4DGS delivers higher reconstruction quality and improved efficiency over current state-of-the-art approaches.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"7176-7188"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148747280","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Dual-Attention Video Representation Learning for Parameter Efficient Text-Video Retrieval","authors":"Ziyi Bian;Yadan Luo;Fangzhi Zhu;Xiangbo Shu;Zheng Zhang","doi":"10.1109/TMM.2026.3673509","DOIUrl":"https://doi.org/10.1109/TMM.2026.3673509","url":null,"abstract":"Recently, significant progress has been made in text-video retrieval by adapting CLIP to the text-video domain. To mitigate the computational overhead of full fine-tuning, current studies have shifted focus toward parameter-efficient fine-tuning strategies, such as Adapters, Prompts, and LoRA. However, most methods generally obtain the global video representation by frame-level aggregation, i.e., pooling operation over [CLS] token in each frame, which fails to capture the local details. Meanwhile, these methods process all visual tokens indiscriminately when learning video representations, overlooking the interference introduced by redundant tokens. To tackle these issues, this paper proposes a  <italic>Dual-Attention Video Representation Learning</i> (DAVRL) framework which synergistically combines dual-attention based global-local interactions with token selection. Specifically, we incorporate a dual-attention global-local interaction paradigm and formulate a progressive cross-frame interaction module, which implement global interactions between well-designed learnable global video tokens and all visual tokens, and progressive local interactions from single-frame to multi-frame scales. Moreover, we propose a semantic-aware token selection module to prune the spatial-temporal redundant tokens, under the guidance of both global and local semantics. As such, our DAVRL framework can learn more informative yet discriminative global video representations while mitigating the interference of redundant tokens. Extensive experiments demonstrate the superiority of our DAVRL on diverse datasets with only <italic>0.32% tunable parameters</i>.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"7204-7216"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148748304","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Generating Imperceptible Perturbations to Attack Human Pose Estimation Networks","authors":"Junlong Mu;Lin Zhao;Di Wang;Chen Gong;Nannan Wang","doi":"10.1109/TMM.2026.3668679","DOIUrl":"https://doi.org/10.1109/TMM.2026.3668679","url":null,"abstract":"Adversarial attacks on deep networks have received significant attention recently. However, most existing research focuses on classification tasks, with limited exploration of adversarial attacks on human pose estimation networks. To bridge this gap, we propose a novel attack framework specifically tailored for human pose estimation by exploiting unique HPE characteristics including heatmap-sensitive localization, joint-influence imbalance, and keypoint-focused perception. Our objective is to substantially diminish object keypoint similarity while introducing minimal perturbations to the image. We have devised a two-stage framework to implement the attack. The first stage involves a gradient attack framework that induces deviation in the adversarial heatmap from the original heatmap by exploiting heatmap-sensitive localization. In the second stage, the perturbations are optimized and restricted to the vicinity of keypoints to make the attacks imperceptible by exploiting joint-influence imbalance and keypoint-focused perception. To achieve this, we incorporate low-frequency constraints to limit the perturbations to high-frequency components and utilize a perceptual color distance metric to control the perturbation’s magnitude. Extensive experimental results on the COCO and MPII datasets demonstrate that our attack can generate adversarial examples with high strength and low detectability.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"6654-6662"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148749114","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"DualSpec: Text-to-Spatial-Audio Generation via Dual-Spectrogram Guided Diffusion Model","authors":"Lei Zhao;Sizhou Chen;Linfeng Feng;Jichao Zhang;Xiao-Lei Zhang;Chi Zhang;Xuelong Li","doi":"10.1109/TMM.2026.3673480","DOIUrl":"https://doi.org/10.1109/TMM.2026.3673480","url":null,"abstract":"Text-to-audio (TTA), which generates audio signals from textual descriptions, has received huge attention in recent years. However, recent works focused on text to monaural audio only. As we know, spatial audio provides more immersive auditory experience than monaural audio, e.g. in virtual reality. To address this issue, we propose a text-to-spatial-audio (TTSA) generation framework named DualSpec. Specifically, it first trains variational autoencoders (VAEs) for extracting the latent acoustic representations from sound event audio. Then, given text that describes sound events and event directions, the proposed method uses the encoder of a pretrained large language model to transform the text into text features. Finally, it trains a diffusion model from the latent acoustic representations and text features for the spatial audio generation. In the inference stage, only the text description is needed to generate spatial audio. Particularly, to improve the synthesis quality and azimuth accuracy of the spatial sound events simultaneously, we propose to use two kinds of acoustic features. One is the Mel spectrograms which is good for improving the synthesis quality, and the other is the short-time Fourier transform spectrograms which is good at improving the azimuth accuracy. We provide a pipeline of constructing spatial audio dataset with text prompts, for the training of the VAEs and diffusion model. We also introduce new spatial-aware evaluation metrics to quantify the azimuth errors of the generated spatial audio recordings. Experimental results demonstrate that the proposed method can generate spatial audio with high directional and event consistency.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"7112-7123"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148749203","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"4DGStream: Variable Bitrate Dynamic Gaussian Splatting Streaming","authors":"Zhicheng Liang;Dayou Zhang;Linfeng Shen;Miao Zhang;Jian Zhang;Bin Ju;Mallesham Dasari;Fangxin Wang;Jiangchuan Liu","doi":"10.1109/TMM.2026.3668629","DOIUrl":"https://doi.org/10.1109/TMM.2026.3668629","url":null,"abstract":"While 3D Gaussian Splatting (3DGS) has revolutionized static scene representation, the extension to dynamic scene, i.e., 3DGS video (GSV), faces challenges related to reconstruction quality, rendering speed, and storage requirements. The substantial data volume of current GSV poses significant hurdles for streaming applications, particularly in the realm of AR, VR and MR. To tackle these challenges, we introduce <monospace>4DGStream</monospace>, a novel framework that integrates an efficient GSV compression method, <monospace>Light4D</monospace>, and a bitrate adaptation streaming strategy, <monospace>QoSmooth</monospace>, to ensure smooth playback while maintaining high visual quality. <monospace>Light4D</monospace> employs a binarization-assisted spatiotemporal deformation network to model the deformation of Gaussian primitive attributes over time, while a spatiotemporal-aware masking module prunes trivial Gaussians, further enhancing long-term reconstruction quality. To reduce storage, <monospace>Light4D</monospace> uses a binary hash grid to model the entropy of attributes for arithmetic coding, with its binary nature allowing efficient entropy modeling via a Bernoulli distribution. These components enable <monospace>Light4D</monospace> to improve the FPS/Storage metric by up to 12.4× over SpacetimeGS and 26.4× over 4DGS on the Neu3D dataset, with performance gains exceeding <inline-formula><tex-math>$3times$</tex-math></inline-formula> orders of magnitude compared to other NeRF-based state-of-the-art (SOTA) methods. Here, FPS/Storage reflects the balance between rendering speed and data storage. Despite significant model size reductions, <monospace>Light4D</monospace> maintains or surpasses the reconstruction quality of 4DGS. Furthermore, <monospace>QoSmooth</monospace> provides effective rate control to enhance playback smoothness, reducing bitrate level switches by 61.6% and increasing time-average utility by 26.2%. All these improvements make <monospace>4DGStream</monospace> highly suited for GSV streaming, improving QoE by 36.7% compared to SOTA methods.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"6877-6891"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148749250","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Lightweight and Unified Framework for Underwater Image Utility Evaluation","authors":"Weiling Chen;Lishi Xu;Honggang Liao;Nanfeng Jiang;Tiesong Zhao","doi":"10.1109/TMM.2026.3668699","DOIUrl":"https://doi.org/10.1109/TMM.2026.3668699","url":null,"abstract":"Transmitting images through narrow-bandwidth underwater acoustic channels presents a significant challenge. However, for many applications, transmitting the entire image is not necessary. Instead, conveying only the critical information is typically sufficient to ensure task success. To address this, we have introduced a preprocessing strategy at the image acquisition stage, which involves down-sampling or feature extraction to distill the data into its essential components. This strategy is guided by utility considerations to ensure that the transmitted data is both concise and informative. However, existing quality evaluation methods are inadequate for providing accurate utility assessments and fail to simultaneously evaluate the utility of both images and features within a unified framework. To bridge this gap in underwater object detection (one of the typical underwater task), we have developed an underwater image utility dataset and introduced the Lightweight and Unified Utility for Underwater Image Evaluation (LU³IE). LU³IE leverages features derived from multiple detection models to enhance generalization. It then refines these features through dimensionality reduction and commonality extraction, and simplifies complexity through the use of knowledge distillation. Experimental results validate the effectiveness and scalability of LU³IE in accurately assessing the utility of underwater images.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"6996-7008"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148749732","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Adapting CLIP for 3D Human Pose Estimation","authors":"Lu Zhou;Yingying Chen;Jinqiao Wang","doi":"10.1109/TMM.2026.3668642","DOIUrl":"https://doi.org/10.1109/TMM.2026.3668642","url":null,"abstract":"Two-stage 3D human pose estimation has garnered remarkable progress, whereas ill-posed issue still poses great deterioration to the performance. During the 2D-3D projection, comprehension of the human topology serves as a significant cue to remove the ambiguities. However, limited feature representation restricts the grasp of human knowledge. In this light, we advocate for enriching the feature representation via integrating information from different sources. Contrastive image-text pretrained models like CLIP demonstrate the capability of generalized representation. Exploiting the strong representation power of these models improves the performance of massive downstream tasks. Inspired by this, we resort to CLIP and delve into the integration of multi-source information with the distillation technique to reduce the location error of the regression task. Firstly, we endeavor to adapt the relation knowledge from CLIP text encoder to the transformer block of joint relation modeling module. Embedding of external language knowledge enriches the feature with the tailored adjuster and conveys the relation information from a quite different aspect. Secondly, we employ the CLIP visual encoder as another source to attain the knowledge from the vision aspect. We transfer the discrete human joints into the sketch image where human skeleton is depicted on a white board. Afterwards, the sketch drawing serves as the input of CLIP visual encoder, leading to a full understanding of human anatomy. We inject this representation into 3D human pose estimation network with the distillation technique where a dual-path distillation module is advanced to retain multimodal information. Extensive experiments on Human3.6M and MPI-INF-3DHP showcase that proposed method achieves competitive results over the previous approaches and improves the location precision, revealing the efficacy of our approach.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"6605-6615"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148749788","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"PReP: Efficient Context-Based Shape Retrieval for Missing Parts","authors":"Vlassis Fotis;Ioannis Romanelis;Georgios Mylonas;Athanasios Kalogeras;Konstantinos Moustakas","doi":"10.1109/TMM.2026.3668613","DOIUrl":"https://doi.org/10.1109/TMM.2026.3668613","url":null,"abstract":"In this paper, we study the problem of shape part retrieval in the point cloud domain. Existing shape retrieval methods typically rely on the presence of a complete query object, but what if the part of interest is missing? We present the Part Retrieval Pipeline (PReP), which combines metric learning techniques with a trained classification model to evaluate the suitability of potential replacement parts from a database, within an application scenario aimed at circular economy. Through a progressively more difficult training procedure, PReP learns to recognize suitable parts based solely on shape context. Owing to its compact parameterization and low computational requirements, it can search a repository of tens of thousands of spare parts in just a few seconds. We also establish an alternative baseline approach for comparison, document the unique challenges associated with this task, and identify key design choices to address them.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"7074-7086"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=11417229","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148749791","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"PRD$^{3}$M: Prototype Representation Dataset Distillation via Diffusion Model","authors":"Weiwei Xiao;Yi Ma;Shaocong Wu;Qiben Shan;Yaowei Wang;Jingyong Su","doi":"10.1109/TMM.2026.3673581","DOIUrl":"https://doi.org/10.1109/TMM.2026.3673581","url":null,"abstract":"Deep neural networks (DNNs) have achieved success, but their reliance on large-scale datasets incurs substantial computational and storage overhead. Dataset distillation (DD) is an effective dataset compression technique that condenses large-scale datasets into small distilled subsets, thereby reducing computational and storage costs. While numerous matching-based DD methods have been proposed, their bi-level optimization in pixel space causes substantial computational costs, thereby limiting their applicability to large-scale datasets and complex DNNs. Recent DD methods leverage pre-trained generative models to generate distilled images, extending to large-scale datasets and improving generalization. However, these methods face the challenge of decoupling the most relevant information to the original dataset from pre-trained generative models. In this paper, we propose a prototype distillation method that utilizes representative and diverse prototypes to generate the distilled images via the stable diffusion. Specifically, we first obtain representative prototypes in the latent space to compress the original dataset by constraining distribution consistency between prototypes and the latent vectors of the original dataset. Subsequently, we ensure the diversity among the representative prototypes through a diversity loss. Finally, we leverage the obtained prototypes and text labels for stable diffusion to generate the distilled images. Compared to existing DD methods, the proposed method reduces both training time and GUP overhead while scaling effectively to large-scale datasets. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art methods on large-scale datasets, while improving across-architecture generalization.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"7279-7289"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148837137","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}