IEEE transactions on pattern analysis and machine intelligence最新文献

Fully-Connected Transformer for Multi-Source Image Fusion 用于多源图像融合的全连接变压器

IEEE transactions on pattern analysis and machine intelligence Pub Date : 2025-02-05 DOI: 10.1109/TPAMI.2024.3523364

Xiao Wu;Zi-Han Cao;Ting-Zhu Huang;Liang-Jian Deng;Jocelyn Chanussot;Gemine Vivone

{"title":"Fully-Connected Transformer for Multi-Source Image Fusion","authors":"Xiao Wu;Zi-Han Cao;Ting-Zhu Huang;Liang-Jian Deng;Jocelyn Chanussot;Gemine Vivone","doi":"10.1109/TPAMI.2024.3523364","DOIUrl":"https://doi.org/10.1109/TPAMI.2024.3523364","url":null,"abstract":"Multi-source image fusion combines the information coming from multiple images into one data, thus improving imaging quality. This topic has aroused great interest in the community. How to integrate information from different sources is still a big challenge, although the existing self-attention based transformer methods can capture spatial and channel similarities. In this paper, we first discuss the mathematical concepts behind the proposed generalized self-attention mechanism, where the existing self-attentions are considered basic forms. The proposed mechanism employs multilinear algebra to drive the development of a novel fully-connected self-attention (FCSA) method to fully exploit local and non-local domain-specific correlations among multi-source images. Moreover, we propose a multi-source image representation embedding it into the FCSA framework as a non-local prior within an optimization problem. Some different fusion problems are unfolded into the proposed fully-connected transformer fusion network (FC-Former). More specifically, the concept of generalized self-attention can promote the potential development of self-attention. Hence, the FC-Former can be viewed as a network model unifying different fusion tasks. Compared with state-of-the-art methods, the proposed FC-Former method exhibits robust and superior performance, showing its capability of faithfully preserving information.","PeriodicalId":94034,"journal":{"name":"IEEE transactions on pattern analysis and machine intelligence","volume":"47 3","pages":"2071-2088"},"PeriodicalIF":0.0,"publicationDate":"2025-02-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143184170","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

RenAIssance: A Survey Into AI Text-to-Image Generation in the Era of Large Model 文艺复兴：大模型时代人工智能文本到图像生成研究

IEEE transactions on pattern analysis and machine intelligence Pub Date : 2024-12-27 DOI: 10.1109/TPAMI.2024.3522305

Fengxiang Bie;Yibo Yang;Zhongzhu Zhou;Adam Ghanem;Minjia Zhang;Zhewei Yao;Xiaoxia Wu;Connor Holmes;Pareesa Golnari;David A. Clifton;Yuxiong He;Dacheng Tao;Shuaiwen Leon Song

{"title":"RenAIssance: A Survey Into AI Text-to-Image Generation in the Era of Large Model","authors":"Fengxiang Bie;Yibo Yang;Zhongzhu Zhou;Adam Ghanem;Minjia Zhang;Zhewei Yao;Xiaoxia Wu;Connor Holmes;Pareesa Golnari;David A. Clifton;Yuxiong He;Dacheng Tao;Shuaiwen Leon Song","doi":"10.1109/TPAMI.2024.3522305","DOIUrl":"10.1109/TPAMI.2024.3522305","url":null,"abstract":"Text-to-image generation (TTI) refers to the usage of models that could process text input and generate high fidelity images based on text descriptions. Text-to-image generation using neural networks could be traced back to the emergence of Generative Adversial Network (GAN), followed by the autoregressive Transformer. Diffusion models are one prominent type of generative model used for the generation of images through the systematic introduction of noises with repeating steps. As an effect of the impressive results of diffusion models on image synthesis, it has been cemented as the major image decoder used by text-to-image models and brought text-to-image generation to the forefront of machine-learning (ML) research. In the era of large models, scaling up model size and the integration with large language models have further improved the performance of TTI models, resulting the generation result nearly indistinguishable from real-world images, revolutionizing the way we retrieval images. Our explorative study has incentivised us to think that there are further ways of scaling text-to-image models with the combination of innovative model architectures and prediction enhancement techniques. We have divided the work of this survey into five main sections wherein we detail the frameworks of major literature in order to delve into the different types of text-to-image generation methods. Following this we provide a detailed comparison and critique of these methods and offer possible pathways of improvement for future work. In the future work, we argue that TTI development could yield impressive productivity improvements for creation, particularly in the context of the AIGC era, and could be extended to more complex tasks such as video generation and 3D generation.","PeriodicalId":94034,"journal":{"name":"IEEE transactions on pattern analysis and machine intelligence","volume":"47 3","pages":"2212-2231"},"PeriodicalIF":0.0,"publicationDate":"2024-12-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142888263","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Natural Adversarial Mask for Face Identity Protection in Physical World 物理世界中面部身份保护的天然对抗面具

IEEE transactions on pattern analysis and machine intelligence Pub Date : 2024-12-26 DOI: 10.1109/TPAMI.2024.3522994

Tianxin Xie;Hu Han;Shiguang Shan;Xilin Chen

{"title":"Natural Adversarial Mask for Face Identity Protection in Physical World","authors":"Tianxin Xie;Hu Han;Shiguang Shan;Xilin Chen","doi":"10.1109/TPAMI.2024.3522994","DOIUrl":"10.1109/TPAMI.2024.3522994","url":null,"abstract":"Facial recognition (FR) technology offers convenience in our daily lives, but it also raises serious privacy issues due to unauthorized FR applications. To protect facial privacy, existing methods have proposed adversarial face examples that can fool FR systems. However, most of these methods work only in the digital domain and do not consider natural physical protections. In this paper, we present NatMask, a 3D-based method for creating natural and realistic adversarial face masks that can preserve facial identity in the physical world. Our method utilizes 3D face reconstruction and differentiable rendering to generate 2D face images with natural-looking facial masks. Moreover, we propose an identity-aware style injection (IASI) method to improve the naturalness and transferability of the mask texture. We evaluate our method on two face datasets to verify its effectiveness in protecting face identity against four state-of-the-art (SOTA) FR models and three commercial FR APIs in both digital and physical domains under black-box impersonation and dodging strategies. Experiments show that our method can generate adversarial masks with superior naturalness and physical realizability to safeguard face identity, outperforming SOTA methods by a large margin.","PeriodicalId":94034,"journal":{"name":"IEEE transactions on pattern analysis and machine intelligence","volume":"47 3","pages":"2089-2106"},"PeriodicalIF":0.0,"publicationDate":"2024-12-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142888264","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Multi-Head Encoding for Extreme Label Classification 极端标签分类的多头编码

IEEE transactions on pattern analysis and machine intelligence Pub Date : 2024-12-25 DOI: 10.1109/TPAMI.2024.3522298

Daojun Liang;Haixia Zhang;Dongfeng Yuan;Minggao Zhang

{"title":"Multi-Head Encoding for Extreme Label Classification","authors":"Daojun Liang;Haixia Zhang;Dongfeng Yuan;Minggao Zhang","doi":"10.1109/TPAMI.2024.3522298","DOIUrl":"10.1109/TPAMI.2024.3522298","url":null,"abstract":"The number of categories of instances in the real world is normally huge, and each instance may contain multiple labels. To distinguish these massive labels utilizing machine learning, eXtreme Label Classification (XLC) has been established. However, as the number of categories increases, the number of parameters and nonlinear operations in the classifier also rises. This results in a Classifier Computational Overload Problem (CCOP). To address this, we propose a Multi-Head Encoding (MHE) mechanism, which replaces the vanilla classifier with a multi-head classifier. During the training process, MHE decomposes extreme labels into the product of multiple short local labels, with each head trained on these local labels. During testing, the predicted labels can be directly calculated from the local predictions of each head. This reduces the computational load geometrically. Then, according to the characteristics of different XLC tasks, e.g., single-label, multi-label, and model pretraining tasks, three MHE-based implementations, i.e., Multi-Head Product, Multi-Head Cascade, and Multi-Head Sampling, are proposed to more effectively cope with CCOP. Moreover, we theoretically demonstrate that MHE can achieve performance approximately equivalent to that of the vanilla classifier by generalizing the low-rank approximation problem from Frobenius-norm to Cross-Entropy. Experimental results show that the proposed methods achieve state-of-the-art performance while significantly streamlining the training and inference processes of XLC tasks.","PeriodicalId":94034,"journal":{"name":"IEEE transactions on pattern analysis and machine intelligence","volume":"47 3","pages":"2199-2211"},"PeriodicalIF":0.0,"publicationDate":"2024-12-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142888268","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Hierarchical Banzhaf Interaction for General Video-Language Representation Learning 通用视频语言表示学习的层次化Banzhaf交互

IEEE transactions on pattern analysis and machine intelligence Pub Date : 2024-12-24 DOI: 10.1109/TPAMI.2024.3522124

Peng Jin;Hao Li;Li Yuan;Shuicheng Yan;Jie Chen

{"title":"Hierarchical Banzhaf Interaction for General Video-Language Representation Learning","authors":"Peng Jin;Hao Li;Li Yuan;Shuicheng Yan;Jie Chen","doi":"10.1109/TPAMI.2024.3522124","DOIUrl":"10.1109/TPAMI.2024.3522124","url":null,"abstract":"Multimodal representation learning, with contrastive learning, plays an important role in the artificial intelligence domain. As an important subfield, video-language representation learning focuses on learning representations using global semantic interactions between pre-defined video-text pairs. However, to enhance and refine such coarse-grained global interactions, more detailed interactions are necessary for fine-grained multimodal learning. In this study, we introduce a new approach that models video-text as game players using multivariate cooperative game theory to handle uncertainty during fine-grained semantic interactions with diverse granularity, flexible combination, and vague intensity. Specifically, we design the Hierarchical Banzhaf Interaction to simulate the fine-grained correspondence between video clips and textual words from hierarchical perspectives. Furthermore, to mitigate the bias in calculations within Banzhaf Interaction, we propose reconstructing the representation through a fusion of single-modal and cross-modal components. This reconstructed representation ensures fine granularity comparable to that of the single-modal representation, while also preserving the adaptive encoding characteristics of cross-modal representation. Additionally, we extend our original structure into a flexible encoder-decoder framework, enabling the model to adapt to various downstream tasks. Extensive experiments on commonly used text-video retrieval, video-question answering, and video captioning benchmarks, with superior performance, validate the effectiveness and generalization of our method.","PeriodicalId":94034,"journal":{"name":"IEEE transactions on pattern analysis and machine intelligence","volume":"47 3","pages":"2125-2139"},"PeriodicalIF":0.0,"publicationDate":"2024-12-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10815073","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142884231","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Self-Supervised Anomaly Detection With Neural Transformations 基于神经变换的自监督异常检测

IEEE transactions on pattern analysis and machine intelligence Pub Date : 2024-12-18 DOI: 10.1109/TPAMI.2024.3519543

Chen Qiu;Marius Kloft;Stephan Mandt;Maja Rudolph

引用次数: 0

Trajectory of Fifths Based on Chroma Subbands Extraction–A New Approach to Music Representation, Analysis, and Classification 基于色度子带提取的五度轨迹--音乐表现、分析和分类的新方法

IEEE transactions on pattern analysis and machine intelligence Pub Date : 2024-12-17 DOI: 10.1109/TPAMI.2024.3519420

Tomasz Lukaszewicz;Dariusz Kania

{"title":"Trajectory of Fifths Based on Chroma Subbands Extraction–A New Approach to Music Representation, Analysis, and Classification","authors":"Tomasz Lukaszewicz;Dariusz Kania","doi":"10.1109/TPAMI.2024.3519420","DOIUrl":"10.1109/TPAMI.2024.3519420","url":null,"abstract":"In this article, we propose a new method of representing and analyzing music audio records. The method is based on the concept of the trajectory of fifths, which was initially developed for the analysis of music represented in MIDI format. To adapt this concept to the needs of audio signal processing, we implement a short-term spectral analysis of a musical piece, followed by a mapping of its subsequent spectral timeframes onto signatures of fifths reflecting relative intensities of sounds associated with each of the 12 pitch classes. Subsequently, the calculation of the characteristic points of the consecutive signatures of fifths enables the creation of the trajectory of fifths. The results of the experiments and statistical analysis conducted in a set of 8996 audio music pieces belonging to 10 genres indicate that this kind of trajectory, just as its MIDI-compliant precursor, is a source of valuable information (i.e., feature coefficients) concerning the harmonic structure of music, which may find use in audio music classification processes.","PeriodicalId":94034,"journal":{"name":"IEEE transactions on pattern analysis and machine intelligence","volume":"47 3","pages":"2157-2169"},"PeriodicalIF":0.0,"publicationDate":"2024-12-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10804652","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142840744","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Fast Semi-Supervised Learning on Large Graphs: An Improved Green-Function Method 大型图上的快速半监督学习：改进的绿色函数方法

IEEE transactions on pattern analysis and machine intelligence Pub Date : 2024-12-17 DOI: 10.1109/TPAMI.2024.3518595

Feiping Nie;Yitao Song;Wei Chang;Rong Wang;Xuelong Li

引用次数: 0

PDPP: Projected Diffusion for Procedure Planning in Instructional Videos PDPP：教学视频中程序规划的预测扩散量

IEEE transactions on pattern analysis and machine intelligence Pub Date : 2024-12-16 DOI: 10.1109/TPAMI.2024.3518762

Hanlin Wang;Yilu Wu;Sheng Guo;Limin Wang

{"title":"PDPP: Projected Diffusion for Procedure Planning in Instructional Videos","authors":"Hanlin Wang;Yilu Wu;Sheng Guo;Limin Wang","doi":"10.1109/TPAMI.2024.3518762","DOIUrl":"10.1109/TPAMI.2024.3518762","url":null,"abstract":"In this paper, we study the problem of procedure planning in instructional videos, which aims to make a plan (i.e. a sequence of actions) given the current visual observation and the desired goal. Previous works cast this as a sequence modeling problem and leverage either intermediate visual observations or language instructions as supervision to make autoregressive planning, resulting in complex learning schemes and expensive annotation costs. To avoid intermediate supervision annotation and error accumulation caused by planning autoregressively, we propose a diffusion-based framework, coined as PDPP (Projected Diffusion model for Procedure Planning), to directly model the whole action sequence distribution with task label as supervision instead. Our core idea is to treat procedure planning as a distribution fitting problem under the given observations, thus transform the planning problem to a sampling process from this distribution during inference. The diffusion-based modeling approach also effectively addresses the uncertainty issue in procedure planning. Based on PDPP, we further apply joint training to our framework to generate plans with varying horizon lengths using a single model and reduce the number of training parameters required. We instantiate our PDPP with three popular diffusion models and investigate a serious of condition-introducing methods in our framework, including condition embeddings, Mixture-of-Experts (MoEs), two-stage prediction and Classifier-Free Guidance strategy. Finally, we apply our PDPP to the Visual Planners for human Assistance (VPA) problem which requires the goal specified in natural language rather than visual observation. We conduct experiments on challenging datasets of different scales and our PDPP model achieves the state-of-the-art performance on multiple metrics, even compared with those strongly-supervised counterparts. These results further demonstratethe effectiveness and generalization ability of our model.","PeriodicalId":94034,"journal":{"name":"IEEE transactions on pattern analysis and machine intelligence","volume":"47 3","pages":"2107-2124"},"PeriodicalIF":0.0,"publicationDate":"2024-12-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142832333","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Continual Learning: Forget-Free Winning Subnetworks for Video Representations 持续学习：用于视频表示的无遗忘制胜子网络

IEEE transactions on pattern analysis and machine intelligence Pub Date : 2024-12-16 DOI: 10.1109/TPAMI.2024.3518588

Haeyong Kang;Jaehong Yoon;Sung Ju Hwang;Chang D. Yoo

{"title":"Continual Learning: Forget-Free Winning Subnetworks for Video Representations","authors":"Haeyong Kang;Jaehong Yoon;Sung Ju Hwang;Chang D. Yoo","doi":"10.1109/TPAMI.2024.3518588","DOIUrl":"10.1109/TPAMI.2024.3518588","url":null,"abstract":"Inspired by the Lottery Ticket Hypothesis (LTH), which highlights the existence of efficient subnetworks within larger, dense networks, a high-performing Winning Subnetwork (WSN) in terms of task performance under appropriate sparsity conditions is considered for various continual learning tasks. It leverages pre-existing weights from dense networks to achieve efficient learning in Task Incremental Learning (TIL) and Task-agnostic Incremental Learning (TaIL) scenarios. In Few-Shot Class Incremental Learning (FSCIL), a variation of WSN referred to as the Soft subnetwork (SoftNet) is designed to prevent overfitting when the data samples are scarce. Furthermore, the sparse reuse of WSN weights is considered for Video Incremental Learning (VIL). The use of Fourier Subneural Operator (FSO) within WSN is considered. It enables compact encoding of videos and identifies reusable subnetworks across varying bandwidths. We have integrated FSO into different architectural frameworks for continual learning, including VIL, TIL, and FSCIL. Our comprehensive experiments demonstrate FSO's effectiveness, significantly improving task performance at various convolutional representational levels. Specifically, FSO enhances higher-layer performance in TIL and FSCIL and lower-layer performance in VIL.","PeriodicalId":94034,"journal":{"name":"IEEE transactions on pattern analysis and machine intelligence","volume":"47 3","pages":"2140-2156"},"PeriodicalIF":0.0,"publicationDate":"2024-12-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142832334","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0