IEEE Transactions on Multimedia最新文献

筛选
英文 中文
Improving the Transferability of Adversarial Examples Through Spatial-Based Color Modification 通过基于空间的颜色修饰提高对抗性示例的可转移性
IF 9.9 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-02-26 DOI: 10.1109/TMM.2026.3668693
Zihan Zhou;Zichi Wang;Xinpeng Zhang;Guorui Feng
{"title":"Improving the Transferability of Adversarial Examples Through Spatial-Based Color Modification","authors":"Zihan Zhou;Zichi Wang;Xinpeng Zhang;Guorui Feng","doi":"10.1109/TMM.2026.3668693","DOIUrl":"https://doi.org/10.1109/TMM.2026.3668693","url":null,"abstract":"Unrestricted adversarial attacks that modify the attributes or content of images have demonstrated significant potential. Among unrestricted adversarial attacks, color modification attacks that targeting image color have increasingly obtained attention from researchers. By perturbing the color information of images, color modification attacks can be more imperceptible to human vision compared with traditional adversarial perturbations constrained by <inline-formula><tex-math>$L_{p}$</tex-math></inline-formula>-norm. Furthermore, the alteration of image attributes significantly disrupts the low-level features extracted by Deep Neural Networks (DNNs), effectively enhancing the transferability. However, current work focus solely on global image attributes, which limit the attack effectiveness. Therefore, this paper proposes Local-Global-Local (LGL), a novel color modification attack framework that incorporates spatial and color information of the image. The color distribution of the image is initialized at first, and then the color filter and spatial masks are used to adjust the global and local color distribution in more detail. Experimental results show that the proposed method achieves superior performance in terms of adversariality and transferability, while also demonstrating robustness against defense methods.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"6663-6674"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148747246","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Standardizing Generative Face Video Compression Using Supplemental Enhancement Information 使用补充增强信息的标准化生成人脸视频压缩
IF 9.9 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-02-27 DOI: 10.1109/TMM.2026.3668615
Bolin Chen;Yan Ye;Jie Chen;Ru-Ling Liao;Shanzhi Yin;Shiqi Wang;Kaifa Yang;Yue Li;Yiling Xu;Ye-Kui Wang;Shiv Gehlot;Guan-Ming Su;Peng Yin;Sean McCarthy;Gary J. Sullivan
{"title":"Standardizing Generative Face Video Compression Using Supplemental Enhancement Information","authors":"Bolin Chen;Yan Ye;Jie Chen;Ru-Ling Liao;Shanzhi Yin;Shiqi Wang;Kaifa Yang;Yue Li;Yiling Xu;Ye-Kui Wang;Shiv Gehlot;Guan-Ming Su;Peng Yin;Sean McCarthy;Gary J. Sullivan","doi":"10.1109/TMM.2026.3668615","DOIUrl":"https://doi.org/10.1109/TMM.2026.3668615","url":null,"abstract":"This paper proposes a Generative Face Video Compression (GFVC) approach using Supplemental Enhancement Information (SEI), where a series of compact spatial and temporal representations of a face video signal (e.g., 2D/3D key-points, facial semantics and compact features) can be coded using SEI messages and inserted into the coded video bitstream. At the time of writing, the proposed GFVC approach using SEI messages has been included into a draft amendment of the Versatile Supplemental Enhancement Information (VSEI) standard by the Joint Video Experts Team (JVET) of ISO/IEC JTC 1/SC 29 and ITU-T SG21, which will be standardized as a new version of ITU-T H.274 <inline-formula><tex-math>$|$</tex-math></inline-formula> ISO/IEC 23002-7. To the best of the authors' knowledge, the JVET work on the proposed SEI-based GFVC approach is the first standardization activity for generative video compression. The proposed SEI approach has not only advanced the reconstruction quality of early-day Model-Based Coding (MBC) via the state-of-the-art generative technique, but also established a new SEI definition for future GFVC applications and deployment. Experimental results illustrate that the proposed SEI-based GFVC approach can achieve remarkable rate-distortion performance compared with the latest Versatile Video Coding (VVC) standard, whilst also potentially enabling a wide variety of functionalities including user-specified animation/filtering and metaverse-related applications.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"6743-6756"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148747333","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
A Survey on Multimodal Recommender Systems: Recent Advances and Future Directions 多模式推荐系统综述:最新进展和未来发展方向
IF 9.9 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-02-27 DOI: 10.1109/TMM.2026.3668620
Jinfeng Xu;Zheyu Chen;Shuo Yang;Jinze Li;Wei Wang;Xiping Hu;Steven Hoi;Edith C. H. Ngai
{"title":"A Survey on Multimodal Recommender Systems: Recent Advances and Future Directions","authors":"Jinfeng Xu;Zheyu Chen;Shuo Yang;Jinze Li;Wei Wang;Xiping Hu;Steven Hoi;Edith C. H. Ngai","doi":"10.1109/TMM.2026.3668620","DOIUrl":"https://doi.org/10.1109/TMM.2026.3668620","url":null,"abstract":"The exponential growth of online information has made it increasingly difficult for users to identify valuable and relevant content. Recommender systems have emerged as a critical solution to this challenge by tailoring content to individual preferences. With the proliferation of diverse multimedia services, human interaction with the digital world has become inherently multimodal. Consequently, recommender systems capable of comprehending and interpreting multimodal information can more effectively align with individual preferences. With its recent surge in research attention, the field of multimodal recommender systems (MRS) still lacks a comprehensive technical survey. Existing surveys suffer from two critical limitations: 1) Insufficient technical depth: Prior works predominantly focus on categorizing and discussing general structure, neglecting rigorous technical analysis of methodologies and architectures. 2) Absence of cutting-edge works: Due to the rapid evolution of AI technologies, current surveys fail to discuss the latest works that adopt the most advanced techniques. To bridge these gaps, this survey conducts a systematic and technical review of advanced MRS works from MRS’s inception to the present. We organize existing works into coherent taxonomies based on their structure and provide in-depth analyses of methodological innovations at each component, including Feature Extraction, Encoder, Multimodal Fusion, and Loss Function. Moreover, we further discuss potential future directions for developing and enhancing MRS. This survey serves as technical guidance for researchers and practitioners, offering insights into the developments, techniques, and future directions of MRS.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"7189-7203"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148748076","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Exploring Inter-Domain Wasserstein Metric for Adaptive Object Detection 探索域间Wasserstein度量用于自适应目标检测
IF 9.9 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-02-27 DOI: 10.1109/TMM.2026.3668648
Yanlong Lin;Ziqian Zhu;Yitian Guo;Zhonghong Ou;Siyuan Yao;Meina Song
{"title":"Exploring Inter-Domain Wasserstein Metric for Adaptive Object Detection","authors":"Yanlong Lin;Ziqian Zhu;Yitian Guo;Zhonghong Ou;Siyuan Yao;Meina Song","doi":"10.1109/TMM.2026.3668648","DOIUrl":"https://doi.org/10.1109/TMM.2026.3668648","url":null,"abstract":"Cross-domain adaptation has achieved significant development in recent years. Nevertheless, the models' performance varies dramatically across different scenarios. How to quantitatively measure inter-domain discrepancies to guide model training remains a challenging problem. Existing methods mainly focus on the trivial pixel-wise differences between the cross-domain images, while they ignore the holistic discrepancies of the domain-specific distributions in various scenarios. Thus their effectivenesses are greatly limited in realistic applications. In this paper, we propose a universal method for measuring inter-domain discrepancies based on Wasserstein distance. It alleviates the impact of intra-domain discrepancies on measurements and enables precise and quantitative representation of inter-domain discrepancies. We further integrate this metric into the generative models, and propose an assistant domain to conduct domain knowledge transfer for cross-domain object detection task. Experiments on three benchmarks validate the effectiveness of the proposed inter-domain measurement metric. Specifically, we achieve 51.8% mAP on CityScapes, 45.3% mAP on Clipart and 58.9% mAP on Watercolor, which are 1.5%, 0.5% and 0.8% higher than the state-of-the-art schemes, respectively.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"6616-6626"},"PeriodicalIF":9.9,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148749297","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Negative Semantic Guided Identity Boundary Construction for Open-World Person Re-Identification 否定语义引导下开放世界人再认同的身份边界构建
IF 9.7 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-12 DOI: 10.1109/TMM.2026.3651057
Xiao-Wen Zhang;Delong Zhang;Yi-Xing Peng;Jingke Meng;Wei-Shi Zheng
{"title":"Negative Semantic Guided Identity Boundary Construction for Open-World Person Re-Identification","authors":"Xiao-Wen Zhang;Delong Zhang;Yi-Xing Peng;Jingke Meng;Wei-Shi Zheng","doi":"10.1109/TMM.2026.3651057","DOIUrl":"https://doi.org/10.1109/TMM.2026.3651057","url":null,"abstract":"Person re-identification (ReID) aims to match person images of the same identity under different camera views. Conventional ReID models mainly consider a closed-world setting where person identities in query and gallery are exactly the same. However, in real-world applications, query identities and gallery identities usually do not exactly contain the same persons. Therefore, open-world ReID has been proposed to match the images of gallery identities (targets) with a large number of non-gallery identities (non-targets). Since some non-targets are quite similar to the targets, the ReID model may make incorrect judgments when verifying these non-targets. To solve this problem, we leverage the impressive cross-modal matching capabilities of the large vision-language model (VLM) to construct <bold>N</b>egative <bold>S</b>emantic guided identity boundaries for each person to develop the open-world <bold>ReID</b> model (NS-ReID). To construct the identity boundary, we propose Virtual Non-target Repulsion that utilizes negative semantics to prompt the ReID model to push virtual non-targets away from the targets. The prompts expressing negative semantics offer a different perspective to guide the training process to avoid contradictory optimization. Moreover, we propose the Dual-Boost Refinement Learning strategy to train learnable identity prompts to capture detailed identity information, which is essential for constructing the identity boundary since the variations among identities are comparatively small. These facilitate the model in constructing the wide identity boundary of each person. Extensive experiments on two benchmark ReID datasets demonstrate that our proposed NS-ReID achieves state-of-the-art performance compared with existing methods.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"2474-2485"},"PeriodicalIF":9.7,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147665329","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
High-Capacity Reversible Data Hiding for JPEG Images Using Ternary Matrix Embedding 使用三元矩阵嵌入的JPEG图像的高容量可逆数据隐藏
IF 9.7 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-12 DOI: 10.1109/TMM.2026.3651010
Mengyao Xiao;Xiaolong Li;Jian Li;Qingchao Jiang;Yao Zhao
{"title":"High-Capacity Reversible Data Hiding for JPEG Images Using Ternary Matrix Embedding","authors":"Mengyao Xiao;Xiaolong Li;Jian Li;Qingchao Jiang;Yao Zhao","doi":"10.1109/TMM.2026.3651010","DOIUrl":"https://doi.org/10.1109/TMM.2026.3651010","url":null,"abstract":"Reversible data hiding (RDH) for JPEG images, particularly those focusing on DCT coefficient modification, has garnered significant attention in recent years. Existing methods primarily select coefficients valued <inline-formula><tex-math>$pm 1$</tex-math></inline-formula> for expansion embedding to avoid significant file size increases caused by modifying zero-valued DCT coefficients. However, zero-valued coefficients, which constitute the majority of DCT coefficients, are more suitable for data embedding to reduce the shift distortion. To efficiently utilize zero-valued coefficients for high-capacity embedding while controlling the file size increment, this paper introduces a novel JPEG RDH method based on ternary matrix embedding, where ternary syndrome trellis codes (STC) is employed on selected zero-valued coefficients to minimize the expansion embedding distortion, and other non-zero-valued coefficients are shifted for reversibility. Furthermore, a novel DCT coefficients measurement strategy is proposed for coefficient selection to further reduce the shift distortion. Extensive experimental validations demonstrate the superiority of the proposed method in various evaluation criteria. Notably, the proposed method achieves more than twice the embedding capacity of some state-of-the-art methods at the same PSNR while maintaining file size increment within acceptable bounds.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"2433-2444"},"PeriodicalIF":9.7,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147665333","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Low-Distortion Steganography in Neural Networks 神经网络中的低失真隐写
IF 9.7 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-06 DOI: 10.1109/TMM.2026.3651012
Yunfei Xie;Zichi Wang
{"title":"Low-Distortion Steganography in Neural Networks","authors":"Yunfei Xie;Zichi Wang","doi":"10.1109/TMM.2026.3651012","DOIUrl":"https://doi.org/10.1109/TMM.2026.3651012","url":null,"abstract":"In steganography, the cover medium takes on various types, including images, videos. Due to the advent of deep learning, neural network models are increasingly employed as the cover medium. In existing approaches, the embedding of secret data results in an obvious degradation of the model's original task performance, such as image classification. This paper proposes a low-distortion steganography method that embeds secret data into neural network models without degrading the model's original performance. The method selectively modifies parameters with minimal correlation to the loss function, ensuring that the model's performance remains unaffected. Furthermore, a suitable modification amplitude is defined to minimize the impact on task performance. Experimental results demonstrate that the proposed method enables low-distortion steganography while significantly improving embedding capacity and security. Notably, a maximum of 3.25 M bits of secret data is successfully embedded into a VGG model on the MNIST dataset, resulting in less than 0.1% accuracy degradation, a significant improvement over previous methods.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"2593-2607"},"PeriodicalIF":9.7,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147665449","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Posture-Movement-Frequency-Enhanced Graph Convolutional Network for Gait Emotion Recognition 基于姿态-运动-频率增强的图卷积网络步态情感识别
IF 9.7 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-12 DOI: 10.1109/TMM.2026.3651117
Faliang Huang;Jiayi Yao;Jiachun Xie;Yihua Ye;Yuping Chen
{"title":"Posture-Movement-Frequency-Enhanced Graph Convolutional Network for Gait Emotion Recognition","authors":"Faliang Huang;Jiayi Yao;Jiachun Xie;Yihua Ye;Yuping Chen","doi":"10.1109/TMM.2026.3651117","DOIUrl":"https://doi.org/10.1109/TMM.2026.3651117","url":null,"abstract":"Recent progress in recognizing emotions through gait analysis has attracted substantial interest. Spatial temporal graph convolutional networks (ST-GCN) have been applied to extract gait features effectively, enabling enhanced emotion recognition. However, existing methods do not account for subtle movement cues that are intricately linked to human emotions, resulting in a lack of representation of emotion intensity. Additionally, these methods fail to consider the cyclic nature of gait, focusing only on local dependencies in the temporal domain. To tackle these limitations, we propose an innovative three-streams graph neural network model PMF-GCN (Posture-Movement-Frequency-enhanced Graph Convolutional Network). First, we introduce for the first time in gait emotion recognition, the integration of movement features from translational and rotational perspectives, combined with frequency-domain data, to capture emotion intensity and global dependencies. Then, we devise a novel adaptive feature fusion mechanism (TR-AFM), which achieves effective extraction of spatial-temporal-specific emotion features from gaits through temporal and spatial attention mechanisms, as well as gated units. Comprehensive experiments on two public datasets show that PMF-GCN achieves leading performance in gait emotion recognition, and achieves state-of-the-art performance.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"3270-3283"},"PeriodicalIF":9.7,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147737060","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
REFusion: A Dual-Stage Network With Repeated Key Feature Embedding for Infrared and Visible Image Fusion 融合:一种重复关键特征嵌入的红外与可见光图像融合双阶段网络
IF 9.7 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-06 DOI: 10.1109/TMM.2026.3651133
Yuman Yuan;Hongyang Bai;Tianyu Deng;Jingyi Wei;Jiajun Lin;Xiaochuan Zhao;Jinzhu Shi
{"title":"REFusion: A Dual-Stage Network With Repeated Key Feature Embedding for Infrared and Visible Image Fusion","authors":"Yuman Yuan;Hongyang Bai;Tianyu Deng;Jingyi Wei;Jiajun Lin;Xiaochuan Zhao;Jinzhu Shi","doi":"10.1109/TMM.2026.3651133","DOIUrl":"https://doi.org/10.1109/TMM.2026.3651133","url":null,"abstract":"In smoke interference scenarios, visible image degradation significantly limits the availability of detailed information for fusion. Moreover, a single fusion network often struggles to effectively extract complementary and key information. To overcome these challenges, we propose a infrared and visible image fusion method tailored for smoke interference, termed the Repeated Key Feature Embedding Fusion Network (REFusion). To fully leverage low-frequency and high-frequency information from source images, we design the network with a pre-fusion stage and a fine-fusion stage. To address the heterogeneity between infrared and visible images, we introduce a Cross-Source Context Association Network (CSCAN). By integrating the self-attention mechanism and a cascading pyramid pooling module, we compute attention interaction weights across modes to capture the most relevant context for fusion. To prevent the fusion from favoring one source disproportionately, we enhance the fine-fusion stage with a Differential Information Reinforcement Module (DIRM) and a three-branch network. This structure establishes long-term dependencies between complementary information and excels in capturing high-frequency details. We further design a tailored loss function to enhance the quality of fused images. Extensive experiments demonstrate that REFusion surpasses nine state-of-the-art methods in both visual quality and quantitative metrics, particularly in reproducing scenes obscured by smoke.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"3174-3189"},"PeriodicalIF":9.7,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147737062","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
FluencyVE: Marrying Temporal-Aware Mamba with Bypass Attention for Video Editing FluencyVE:结合时间感知曼巴与旁路注意视频编辑
IF 9.7 1区 计算机科学
IEEE Transactions on Multimedia Pub Date : 2026-01-01 Epub Date: 2026-01-06 DOI: 10.1109/TMM.2026.3651097
Mingshu Cai;Yixuan Li;Osamu Yoshie;Yuya Ieiri
{"title":"FluencyVE: Marrying Temporal-Aware Mamba with Bypass Attention for Video Editing","authors":"Mingshu Cai;Yixuan Li;Osamu Yoshie;Yuya Ieiri","doi":"10.1109/TMM.2026.3651097","DOIUrl":"https://doi.org/10.1109/TMM.2026.3651097","url":null,"abstract":"Large-scale text-to-image diffusion models have achieved unprecedented success in image generation and editing. However, extending this success to video editing remains challenging. Recent video editing efforts have adapted pretrained text-to-image models by adding temporal attention mechanisms to handle video tasks. Unfortunately, these methods continue to suffer from temporal inconsistency issues and high computational overheads. In this study, we propose FluencyVE, which is a simple yet effective one-shot video editing approach. FluencyVE integrates the linear time-series module, Mamba, into a video editing model based on pretrained Stable Diffusion models, replacing the temporal attention layer. This enables global frame-level attention while reducing the computational costs. In addition, we employ low-rank approximation matrices to replace the query and key weight matrices in the causal attention, and use a weighted averaging technique during training to update the attention scores. This approach significantly preserves the generative power of the text-to-image model while effectively reducing the computational burden. Experiments and analyses demonstrate promising results in editing various attributes, subjects, and locations in real-world videos.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"28 ","pages":"3202-3213"},"PeriodicalIF":9.7,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=11329476","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147737119","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
相关产品
×
本文献相关产品
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信
小红书