Intelligent Multi-Modal Sensing-Communication Integration: Synesthesia of Machines

IF 34.4 1区计算机科学 Q1 COMPUTER SCIENCE, INFORMATION SYSTEMS

IEEE Communications Surveys and Tutorials Pub Date : 2023-11-28 DOI:10.1109/COMST.2023.3336917

Xiang Cheng;Haotian Zhang;Jianan Zhang;Shijian Gao;Sijiang Li;Ziwei Huang;Lu Bai;Zonghui Yang;Xinhu Zheng;Liuqing Yang

{"title":"Intelligent Multi-Modal Sensing-Communication Integration: Synesthesia of Machines","authors":"Xiang Cheng;Haotian Zhang;Jianan Zhang;Shijian Gao;Sijiang Li;Ziwei Huang;Lu Bai;Zonghui Yang;Xinhu Zheng;Liuqing Yang","doi":"10.1109/COMST.2023.3336917","DOIUrl":null,"url":null,"abstract":"In the era of sixth-generation (6G) wireless communications, integrated sensing and communications (ISAC) is recognized as a promising solution to upgrade the physical system by endowing wireless communications with sensing capability. Existing ISAC is mainly oriented to static scenarios with radio-frequency (RF) sensors being the primary participants, thus lacking a comprehensive environment feature characterization and facing a severe performance bottleneck in dynamic environments. To date, extensive surveys on ISAC have been conducted but are limited to summarizing RF-based radar sensing. Currently, some research efforts have been devoted to exploring multi-modal sensing-communication integration but still lack a comprehensive review. To fill the gap, we embark on an initial endeavor with the goal of establishing a unified framework of intelligent multi-modal sensing-communication integration by generalizing the concept of ISAC and providing a comprehensive review under this framework. Inspired by the human synesthesia, the so-termed Synesthesia of Machines (SoM) gives the clearest cognition of such an intelligent integration and details its paradigm for the first time. We commence by justifying the necessity and potential of the new paradigm. Subsequently, we offer a rigorous definition of SoM and zoom into the detailed paradigm, which is summarized as three operational modes realizing the integration. To facilitate SoM research, we overview the prerequisite of SoM research, that is, mixed multi-modal (MMM) datasets, and introduce our work. Built upon the MMM datasets, we introduce the mapping relationships between multi-modal sensing and communications, and discuss how channel modeling can be customized to support the exploration of such relationships. Afterward, aiming at giving a comprehensive survey on the current research status of multi-modal sensing-communication integration, we cover the technological review on SoM-enhance-based and SoM-concert-based applications in transceiver design and environment sensing. To corroborate the rationality and superiority of SoM, we also present simulation results related to dual-function waveform and predictive beamforming design tailored for dynamic scenarios. Finally, we propose some open issues and potential directions to inspire future research efforts on SoM.","PeriodicalId":55029,"journal":{"name":"IEEE Communications Surveys and Tutorials","volume":"26 1","pages":"258-301"},"PeriodicalIF":34.4000,"publicationDate":"2023-11-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Communications Surveys and Tutorials","FirstCategoryId":"94","ListUrlMain":"https://ieeexplore.ieee.org/document/10330577/","RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}

引用次数: 0

Abstract

In the era of sixth-generation (6G) wireless communications, integrated sensing and communications (ISAC) is recognized as a promising solution to upgrade the physical system by endowing wireless communications with sensing capability. Existing ISAC is mainly oriented to static scenarios with radio-frequency (RF) sensors being the primary participants, thus lacking a comprehensive environment feature characterization and facing a severe performance bottleneck in dynamic environments. To date, extensive surveys on ISAC have been conducted but are limited to summarizing RF-based radar sensing. Currently, some research efforts have been devoted to exploring multi-modal sensing-communication integration but still lack a comprehensive review. To fill the gap, we embark on an initial endeavor with the goal of establishing a unified framework of intelligent multi-modal sensing-communication integration by generalizing the concept of ISAC and providing a comprehensive review under this framework. Inspired by the human synesthesia, the so-termed Synesthesia of Machines (SoM) gives the clearest cognition of such an intelligent integration and details its paradigm for the first time. We commence by justifying the necessity and potential of the new paradigm. Subsequently, we offer a rigorous definition of SoM and zoom into the detailed paradigm, which is summarized as three operational modes realizing the integration. To facilitate SoM research, we overview the prerequisite of SoM research, that is, mixed multi-modal (MMM) datasets, and introduce our work. Built upon the MMM datasets, we introduce the mapping relationships between multi-modal sensing and communications, and discuss how channel modeling can be customized to support the exploration of such relationships. Afterward, aiming at giving a comprehensive survey on the current research status of multi-modal sensing-communication integration, we cover the technological review on SoM-enhance-based and SoM-concert-based applications in transceiver design and environment sensing. To corroborate the rationality and superiority of SoM, we also present simulation results related to dual-function waveform and predictive beamforming design tailored for dynamic scenarios. Finally, we propose some open issues and potential directions to inspire future research efforts on SoM.

查看原文本刊更多论文

智能多模式传感-通信集成：机器的共鸣

在第六代（6G）无线通信时代，综合传感与通信（ISAC）被认为是通过赋予无线通信以传感能力来升级物理系统的一种有前途的解决方案。现有的 ISAC 主要面向静态场景，以射频（RF）传感器为主要参与者，因此缺乏全面的环境特征描述，在动态环境中面临严重的性能瓶颈。迄今为止，关于 ISAC 的研究已经进行了大量调查，但仅限于对基于射频的雷达传感进行总结。目前，一些研究工作致力于探索多模式传感-通信集成，但仍缺乏全面的综述。为了填补这一空白，我们开始了初步的尝试，目标是通过概括 ISAC 的概念，建立智能多模态传感-通信集成的统一框架，并在此框架下提供全面的综述。受人类联觉的启发，所谓的机器联觉（SoM）对这种智能集成给出了最清晰的认知，并首次详细介绍了其范式。我们首先论证了新范式的必要性和潜力。随后，我们给出了 SoM 的严格定义，并详细介绍了其范式，概括为实现集成的三种运行模式。为了促进 SoM 研究，我们概述了 SoM 研究的先决条件，即混合多模态（MMM）数据集，并介绍了我们的工作。在多模式混合数据集的基础上，我们介绍了多模式传感与通信之间的映射关系，并讨论了如何定制信道建模以支持对这种关系的探索。随后，为了全面考察多模态传感与通信集成的研究现状，我们对收发器设计和环境感知中基于SoM增强和SoM增强的应用进行了技术综述。为了证实 SoM 的合理性和优越性，我们还介绍了针对动态场景的双功能波形和预测波束成形设计的相关仿真结果。最后，我们提出了一些有待解决的问题和潜在的研究方向，以激励未来的 SoM 研究工作。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

IEEE Communications Surveys and Tutorials COMPUTER SCIENCE, INFORMATION SYSTEMS-TELECOMMUNICATIONS

CiteScore

80.20

自引率

2.50%

发文量

审稿时长

6 months

期刊介绍： IEEE Communications Surveys & Tutorials is an online journal published by the IEEE Communications Society for tutorials and surveys covering all aspects of the communications field. Telecommunications technology is progressing at a rapid pace, and the IEEE Communications Society is committed to providing researchers and other professionals the information and tools to stay abreast. IEEE Communications Surveys and Tutorials focuses on integrating and adding understanding to the existing literature on communications, putting results in context. Whether searching for in-depth information about a familiar area or an introduction into a new area, IEEE Communications Surveys & Tutorials aims to be the premier source of peer-reviewed, comprehensive tutorials and surveys, and pointers to further sources. IEEE Communications Surveys & Tutorials publishes only articles exclusively written for IEEE Communications Surveys & Tutorials and go through a rigorous review process before their publication in the quarterly issues. A tutorial article in the IEEE Communications Surveys & Tutorials should be designed to help the reader to become familiar with and learn something specific about a chosen topic. In contrast, the term survey, as applied here, is defined to mean a survey of the literature. A survey article in IEEE Communications Surveys & Tutorials should provide a comprehensive review of developments in a selected area, covering its development from its inception to its current state and beyond, and illustrating its development through liberal citations from the literature. Both tutorials and surveys should be tutorial in nature and should be written in a style comprehensible to readers outside the specialty of the article.