融合惯性和高分辨率声学数据的隐私保护人体活动识别

IF 5.6 2区 工程技术 Q1 ENGINEERING, ELECTRICAL & ELECTRONIC
Zhe Yang;Ying Zhang;Yanjun Li;Linchong Huang;Ping Hu;Yuexiang Lin
{"title":"融合惯性和高分辨率声学数据的隐私保护人体活动识别","authors":"Zhe Yang;Ying Zhang;Yanjun Li;Linchong Huang;Ping Hu;Yuexiang Lin","doi":"10.1109/TIM.2025.3565250","DOIUrl":null,"url":null,"abstract":"Multimodal human activity recognition (HAR) offers significant advantages over single-modality approaches, particularly in the recently discussed fusion of inertial and acoustic data. However, audio information often contains sensitive personal information. While some studies have focused on audio privacy protection, speech frequencies (below 8 kHz) can still be potentially reconstructed using deep learning techniques. This article presents a novel approach to protect audio privacy in multimodal HAR by utilizing low-cost microphones to extract high-resolution (Hi-res) audio and filtering sensitive information at both nonspeech (<inline-formula> <tex-math>$8\\sim 96$ </tex-math></inline-formula> kHz) and inaudible (<inline-formula> <tex-math>$20\\sim 96$ </tex-math></inline-formula> kHz) levels. We collected a dataset of 20 comprehensive daily activities from 15 participants using custom hardware, with ground truth built from video evidence. Building on this foundation, this article proposes a new hybrid-attention-based HAR method, which leverages self-attention (SA) for extracting salient features in both the temporal and latent space domains, as well as cross-attention (CA) for exploring intermodal relationships. According to the evaluation on the collected dataset, the proposed method demonstrates significant performance improvements over single-modality approaches and outperforms common direct concatenation fusion methods. In addition, inaudible ultrasonic frequencies have demonstrated the ability to differentiate certain activities, making them effective for multimodal fusion in scenarios with strict privacy requirements.","PeriodicalId":13341,"journal":{"name":"IEEE Transactions on Instrumentation and Measurement","volume":"74 ","pages":"1-20"},"PeriodicalIF":5.6000,"publicationDate":"2025-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Fusion of Inertial and High-Resolution Acoustic Data for Privacy-Preserving Human Activity Recognition\",\"authors\":\"Zhe Yang;Ying Zhang;Yanjun Li;Linchong Huang;Ping Hu;Yuexiang Lin\",\"doi\":\"10.1109/TIM.2025.3565250\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Multimodal human activity recognition (HAR) offers significant advantages over single-modality approaches, particularly in the recently discussed fusion of inertial and acoustic data. However, audio information often contains sensitive personal information. While some studies have focused on audio privacy protection, speech frequencies (below 8 kHz) can still be potentially reconstructed using deep learning techniques. This article presents a novel approach to protect audio privacy in multimodal HAR by utilizing low-cost microphones to extract high-resolution (Hi-res) audio and filtering sensitive information at both nonspeech (<inline-formula> <tex-math>$8\\\\sim 96$ </tex-math></inline-formula> kHz) and inaudible (<inline-formula> <tex-math>$20\\\\sim 96$ </tex-math></inline-formula> kHz) levels. We collected a dataset of 20 comprehensive daily activities from 15 participants using custom hardware, with ground truth built from video evidence. Building on this foundation, this article proposes a new hybrid-attention-based HAR method, which leverages self-attention (SA) for extracting salient features in both the temporal and latent space domains, as well as cross-attention (CA) for exploring intermodal relationships. According to the evaluation on the collected dataset, the proposed method demonstrates significant performance improvements over single-modality approaches and outperforms common direct concatenation fusion methods. In addition, inaudible ultrasonic frequencies have demonstrated the ability to differentiate certain activities, making them effective for multimodal fusion in scenarios with strict privacy requirements.\",\"PeriodicalId\":13341,\"journal\":{\"name\":\"IEEE Transactions on Instrumentation and Measurement\",\"volume\":\"74 \",\"pages\":\"1-20\"},\"PeriodicalIF\":5.6000,\"publicationDate\":\"2025-04-30\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"IEEE Transactions on Instrumentation and Measurement\",\"FirstCategoryId\":\"5\",\"ListUrlMain\":\"https://ieeexplore.ieee.org/document/10980212/\",\"RegionNum\":2,\"RegionCategory\":\"工程技术\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"ENGINEERING, ELECTRICAL & ELECTRONIC\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Transactions on Instrumentation and Measurement","FirstCategoryId":"5","ListUrlMain":"https://ieeexplore.ieee.org/document/10980212/","RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ENGINEERING, ELECTRICAL & ELECTRONIC","Score":null,"Total":0}
引用次数: 0

摘要

与单模态方法相比,多模态人类活动识别(HAR)具有显著的优势,特别是在最近讨论的惯性和声学数据融合方面。然而,音频信息往往包含敏感的个人信息。虽然一些研究集中在音频隐私保护上,但语音频率(低于8 kHz)仍然可以使用深度学习技术进行重建。本文提出了一种在多模态HAR中保护音频隐私的新方法,该方法利用低成本麦克风提取高分辨率(Hi-res)音频,并在非语音($8\sim 96$ kHz)和不可听($20\sim 96$ kHz)级别过滤敏感信息。我们使用自定义硬件从15名参与者那里收集了20项综合日常活动的数据集,并根据视频证据建立了地面真相。在此基础上,本文提出了一种新的基于混合注意的HAR方法,该方法利用自注意(SA)来提取时间和潜在空间域中的显著特征,并利用交叉注意(CA)来探索多式联运关系。通过对收集到的数据集的评估,该方法比单模态方法的性能有了显著的提高,并且优于一般的直接连接融合方法。此外,不可听的超声波频率已经证明了区分某些活动的能力,使其在具有严格隐私要求的情况下有效地进行多模式融合。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
Fusion of Inertial and High-Resolution Acoustic Data for Privacy-Preserving Human Activity Recognition
Multimodal human activity recognition (HAR) offers significant advantages over single-modality approaches, particularly in the recently discussed fusion of inertial and acoustic data. However, audio information often contains sensitive personal information. While some studies have focused on audio privacy protection, speech frequencies (below 8 kHz) can still be potentially reconstructed using deep learning techniques. This article presents a novel approach to protect audio privacy in multimodal HAR by utilizing low-cost microphones to extract high-resolution (Hi-res) audio and filtering sensitive information at both nonspeech ( $8\sim 96$ kHz) and inaudible ( $20\sim 96$ kHz) levels. We collected a dataset of 20 comprehensive daily activities from 15 participants using custom hardware, with ground truth built from video evidence. Building on this foundation, this article proposes a new hybrid-attention-based HAR method, which leverages self-attention (SA) for extracting salient features in both the temporal and latent space domains, as well as cross-attention (CA) for exploring intermodal relationships. According to the evaluation on the collected dataset, the proposed method demonstrates significant performance improvements over single-modality approaches and outperforms common direct concatenation fusion methods. In addition, inaudible ultrasonic frequencies have demonstrated the ability to differentiate certain activities, making them effective for multimodal fusion in scenarios with strict privacy requirements.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
IEEE Transactions on Instrumentation and Measurement
IEEE Transactions on Instrumentation and Measurement 工程技术-工程:电子与电气
CiteScore
9.00
自引率
23.20%
发文量
1294
审稿时长
3.9 months
期刊介绍: Papers are sought that address innovative solutions to the development and use of electrical and electronic instruments and equipment to measure, monitor and/or record physical phenomena for the purpose of advancing measurement science, methods, functionality and applications. The scope of these papers may encompass: (1) theory, methodology, and practice of measurement; (2) design, development and evaluation of instrumentation and measurement systems and components used in generating, acquiring, conditioning and processing signals; (3) analysis, representation, display, and preservation of the information obtained from a set of measurements; and (4) scientific and technical support to establishment and maintenance of technical standards in the field of Instrumentation and Measurement.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信