Muhammad Ahmad;Muhammad Usama;Manuel Mazzara;Salvatore Distefano;Hamad Ahmed Altuwaijri;Silvia Liberata Ullo
{"title":"融合调谐叉结构中的变换器,实现跨不同样本的高光谱图像分类","authors":"Muhammad Ahmad;Muhammad Usama;Manuel Mazzara;Salvatore Distefano;Hamad Ahmed Altuwaijri;Silvia Liberata Ullo","doi":"10.1109/JSTARS.2024.3465831","DOIUrl":null,"url":null,"abstract":"The 3-D swin transformer (3DST) and spatial–spectral transformer (SST) each excel in capturing distinct aspects of image information: the 3DST with hierarchical attention and window-based processing, and the SST with self-attention mechanisms for long-range dependencies. However, applying them independently reveals the following limitations: the 3DST struggles with spectral information, while the SST lacks in capturing fine spatial details. In this article, we propose a novel tuning fork fusion approach to overcome these shortcomings, integrating the 3DST and SST to enhance the hyperspectral image (HSI) classification (HSIC). Our method integrates the hierarchical attention mechanism from the 3DST with the long-range dependence modeling of the SST. This combination refines spatial and spectral information representation and merges insights from both transformers at a fine-grained level. By emphasizing the fusion of attention mechanisms from both architectures, our approach significantly enhances the model's ability to capture complex spatial–spectral relationships, resulting in improved HSIC accuracy. In addition, we highlight the importance of disjoint training, validation, and test samples to enhance model generalization. Experimentation on benchmark HSI datasets demonstrates the superiority of our fusion approach over other state-of-the-art methods and standalone transformers. The source code has been developed from scratch and will be made public upon acceptance.","PeriodicalId":13116,"journal":{"name":"IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing","volume":null,"pages":null},"PeriodicalIF":4.7000,"publicationDate":"2024-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10685113","citationCount":"0","resultStr":"{\"title\":\"Fusing Transformers in a Tuning Fork Structure for Hyperspectral Image Classification Across Disjoint Samples\",\"authors\":\"Muhammad Ahmad;Muhammad Usama;Manuel Mazzara;Salvatore Distefano;Hamad Ahmed Altuwaijri;Silvia Liberata Ullo\",\"doi\":\"10.1109/JSTARS.2024.3465831\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"The 3-D swin transformer (3DST) and spatial–spectral transformer (SST) each excel in capturing distinct aspects of image information: the 3DST with hierarchical attention and window-based processing, and the SST with self-attention mechanisms for long-range dependencies. However, applying them independently reveals the following limitations: the 3DST struggles with spectral information, while the SST lacks in capturing fine spatial details. In this article, we propose a novel tuning fork fusion approach to overcome these shortcomings, integrating the 3DST and SST to enhance the hyperspectral image (HSI) classification (HSIC). Our method integrates the hierarchical attention mechanism from the 3DST with the long-range dependence modeling of the SST. This combination refines spatial and spectral information representation and merges insights from both transformers at a fine-grained level. By emphasizing the fusion of attention mechanisms from both architectures, our approach significantly enhances the model's ability to capture complex spatial–spectral relationships, resulting in improved HSIC accuracy. In addition, we highlight the importance of disjoint training, validation, and test samples to enhance model generalization. Experimentation on benchmark HSI datasets demonstrates the superiority of our fusion approach over other state-of-the-art methods and standalone transformers. The source code has been developed from scratch and will be made public upon acceptance.\",\"PeriodicalId\":13116,\"journal\":{\"name\":\"IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":4.7000,\"publicationDate\":\"2024-09-23\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10685113\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing\",\"FirstCategoryId\":\"5\",\"ListUrlMain\":\"https://ieeexplore.ieee.org/document/10685113/\",\"RegionNum\":2,\"RegionCategory\":\"地球科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"ENGINEERING, ELECTRICAL & ELECTRONIC\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing","FirstCategoryId":"5","ListUrlMain":"https://ieeexplore.ieee.org/document/10685113/","RegionNum":2,"RegionCategory":"地球科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ENGINEERING, ELECTRICAL & ELECTRONIC","Score":null,"Total":0}
Fusing Transformers in a Tuning Fork Structure for Hyperspectral Image Classification Across Disjoint Samples
The 3-D swin transformer (3DST) and spatial–spectral transformer (SST) each excel in capturing distinct aspects of image information: the 3DST with hierarchical attention and window-based processing, and the SST with self-attention mechanisms for long-range dependencies. However, applying them independently reveals the following limitations: the 3DST struggles with spectral information, while the SST lacks in capturing fine spatial details. In this article, we propose a novel tuning fork fusion approach to overcome these shortcomings, integrating the 3DST and SST to enhance the hyperspectral image (HSI) classification (HSIC). Our method integrates the hierarchical attention mechanism from the 3DST with the long-range dependence modeling of the SST. This combination refines spatial and spectral information representation and merges insights from both transformers at a fine-grained level. By emphasizing the fusion of attention mechanisms from both architectures, our approach significantly enhances the model's ability to capture complex spatial–spectral relationships, resulting in improved HSIC accuracy. In addition, we highlight the importance of disjoint training, validation, and test samples to enhance model generalization. Experimentation on benchmark HSI datasets demonstrates the superiority of our fusion approach over other state-of-the-art methods and standalone transformers. The source code has been developed from scratch and will be made public upon acceptance.
期刊介绍:
The IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing addresses the growing field of applications in Earth observations and remote sensing, and also provides a venue for the rapidly expanding special issues that are being sponsored by the IEEE Geosciences and Remote Sensing Society. The journal draws upon the experience of the highly successful “IEEE Transactions on Geoscience and Remote Sensing” and provide a complementary medium for the wide range of topics in applied earth observations. The ‘Applications’ areas encompasses the societal benefit areas of the Global Earth Observations Systems of Systems (GEOSS) program. Through deliberations over two years, ministers from 50 countries agreed to identify nine areas where Earth observation could positively impact the quality of life and health of their respective countries. Some of these are areas not traditionally addressed in the IEEE context. These include biodiversity, health and climate. Yet it is the skill sets of IEEE members, in areas such as observations, communications, computers, signal processing, standards and ocean engineering, that form the technical underpinnings of GEOSS. Thus, the Journal attracts a broad range of interests that serves both present members in new ways and expands the IEEE visibility into new areas.