Developing an intelligent IoT-enabled wearable multimodal biosensing device and cloud-based digital dashboard for real-time and comprehensive health, physiological, emotional, and cognitive monitoring using multi-sensor fusion technologies

IF 4.1 3区工程技术 Q2 ENGINEERING, ELECTRICAL & ELECTRONIC

Sensors and Actuators A-physical Pub Date : 2024-11-18 DOI:10.1016/j.sna.2024.116074

Rayan H. Assaad , Mohsen Mohammadi , Oscar Poudel

{"title":"Developing an intelligent IoT-enabled wearable multimodal biosensing device and cloud-based digital dashboard for real-time and comprehensive health, physiological, emotional, and cognitive monitoring using multi-sensor fusion technologies","authors":"Rayan H. Assaad , Mohsen Mohammadi , Oscar Poudel","doi":"10.1016/j.sna.2024.116074","DOIUrl":null,"url":null,"abstract":"<div><div>A variety of biosensors have been recently introduced as wearable devices to collect physiological data, with applications ranging from personalized medicine and point-of-care diagnostics to home and fitness monitoring, among others, garnering substantial interest. This interest has been fueled by the increasing demand for ubiquitous, continuous, and pervasive vital signs monitoring, coupled with advancements in biosensor technology and IoT-enabled capabilities. Existing research studies have only relied on a limited number of health- and physiological-related indicators (thus, do not offer a comprehensive health monitoring and assessment system) due to the technical difficulties to integrate multiple sensors. In fact, the issues of multimodality, heterogeneity, and complexity of data as well as the interoperability among sensors make it challenging to seamlessly integrate multiple sensors into one system. This study overcame these technical challenges by leveraging multi-sensor fusion capabilities to develop an intelligent, IoT-enabled wearable multi-modal biosensing device and cloud-based digital dashboard for real-time, comprehensive health, physiological, emotional, and cognitive monitoring. First, 18 different health- and physiological-related indicators were identified. Second, 14 different sensors were used to acquire the entire data for the 18 different indicators using a hardware sensing system designed using four ESP32 microcontroller boards integrated with Wi-Fi and Bluetooth connectivity by fusing the various data from the 14 different sensors. Third, the designed system was developed as a wearable device that can be installed on the hip as well as the right and left feet using 3D printed parts. Fourth, a web-based digital dashboard was created onan edge computing server that was hosted on a microprocessor to instantly publish the data, and a graphical user interface (GUI) was developed to provide intuitive and real-time visualization of the various health-related indicators using the Django and JavaScript-based React.js web development frameworks. The accuracy of the developed IoT-enabled biosensing system was tested and validated by benchmarking and comparing the obtained results from the proposed system with those aquired from various commercially used sensors. The validation outcomes reflected that the proposed system achieved an accuracy of more than 90 % for most of the 18 considered indicators and an accuracy greater than 85 % for all indicators. This study adds to the body of knowledge by being the first research capable of reporting the following 18 indicators into a single biosensing system in real-time: Electrocardiogram (ECG or EKG), Electroencephalogram (EEG), Electrooculogram (EOG), Electromyography (EMG), Photoplethysmography (PPG), heart rate (HR), heart rate variability (HRV), respiratory rate (RR), skin temperature (ST), skin humidity (SH), blood glucose level (BGL), blood pressure (BP), oxygen saturation (SpO2), body weight pressure (BWP), body motion (BM), electrodermal activity (EDA), galvanic skin response (GSR), and skin conductance responses (SCR). The proposed system provides rich information on various vital signs and could be used for a wide window of applications, including monitoring and assessing health status; emotional and arousal status; mental and cognitive status; behavioral, physical, and attention status; and physiological status. The developed system is not specific to a particular industry but rather could be used for any sector of interest. This paper lays the ground to significant advancements in wearable sensor technology, data visualization techniques, and health monitoring practices.</div></div>","PeriodicalId":21689,"journal":{"name":"Sensors and Actuators A-physical","volume":"381 ","pages":"Article 116074"},"PeriodicalIF":4.1000,"publicationDate":"2024-11-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Sensors and Actuators A-physical","FirstCategoryId":"5","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0924424724010689","RegionNum":3,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"ENGINEERING, ELECTRICAL & ELECTRONIC","Score":null,"Total":0}

引用次数: 0

Abstract

A variety of biosensors have been recently introduced as wearable devices to collect physiological data, with applications ranging from personalized medicine and point-of-care diagnostics to home and fitness monitoring, among others, garnering substantial interest. This interest has been fueled by the increasing demand for ubiquitous, continuous, and pervasive vital signs monitoring, coupled with advancements in biosensor technology and IoT-enabled capabilities. Existing research studies have only relied on a limited number of health- and physiological-related indicators (thus, do not offer a comprehensive health monitoring and assessment system) due to the technical difficulties to integrate multiple sensors. In fact, the issues of multimodality, heterogeneity, and complexity of data as well as the interoperability among sensors make it challenging to seamlessly integrate multiple sensors into one system. This study overcame these technical challenges by leveraging multi-sensor fusion capabilities to develop an intelligent, IoT-enabled wearable multi-modal biosensing device and cloud-based digital dashboard for real-time, comprehensive health, physiological, emotional, and cognitive monitoring. First, 18 different health- and physiological-related indicators were identified. Second, 14 different sensors were used to acquire the entire data for the 18 different indicators using a hardware sensing system designed using four ESP32 microcontroller boards integrated with Wi-Fi and Bluetooth connectivity by fusing the various data from the 14 different sensors. Third, the designed system was developed as a wearable device that can be installed on the hip as well as the right and left feet using 3D printed parts. Fourth, a web-based digital dashboard was created onan edge computing server that was hosted on a microprocessor to instantly publish the data, and a graphical user interface (GUI) was developed to provide intuitive and real-time visualization of the various health-related indicators using the Django and JavaScript-based React.js web development frameworks. The accuracy of the developed IoT-enabled biosensing system was tested and validated by benchmarking and comparing the obtained results from the proposed system with those aquired from various commercially used sensors. The validation outcomes reflected that the proposed system achieved an accuracy of more than 90 % for most of the 18 considered indicators and an accuracy greater than 85 % for all indicators. This study adds to the body of knowledge by being the first research capable of reporting the following 18 indicators into a single biosensing system in real-time: Electrocardiogram (ECG or EKG), Electroencephalogram (EEG), Electrooculogram (EOG), Electromyography (EMG), Photoplethysmography (PPG), heart rate (HR), heart rate variability (HRV), respiratory rate (RR), skin temperature (ST), skin humidity (SH), blood glucose level (BGL), blood pressure (BP), oxygen saturation (SpO2), body weight pressure (BWP), body motion (BM), electrodermal activity (EDA), galvanic skin response (GSR), and skin conductance responses (SCR). The proposed system provides rich information on various vital signs and could be used for a wide window of applications, including monitoring and assessing health status; emotional and arousal status; mental and cognitive status; behavioral, physical, and attention status; and physiological status. The developed system is not specific to a particular industry but rather could be used for any sector of interest. This paper lays the ground to significant advancements in wearable sensor technology, data visualization techniques, and health monitoring practices.

查看原文本刊更多论文

利用多传感器融合技术开发智能物联网可穿戴多模态生物传感设备和基于云的数字仪表板，用于实时和全面的健康、生理、情感和认知监测

最近，各种生物传感器作为收集生理数据的可穿戴设备问世，其应用范围从个性化医疗和护理点诊断到家庭和健身监测等，引起了人们的极大兴趣。人们对无处不在、连续和普遍的生命体征监测的需求日益增长，加上生物传感器技术和物联网功能的进步，激发了人们的兴趣。由于集成多个传感器存在技术难度，现有的研究仅依赖于数量有限的健康和生理相关指标（因此无法提供全面的健康监测和评估系统）。事实上，数据的多模态性、异质性和复杂性以及传感器之间的互操作性等问题使得将多个传感器无缝集成到一个系统中具有挑战性。本研究利用多传感器融合功能，开发了一种智能的、支持物联网的可穿戴多模态生物传感设备和基于云的数字仪表板，用于实时、全面的健康、生理、情感和认知监测，从而克服了这些技术挑战。首先，确定了 18 种不同的健康和生理相关指标。其次，使用 14 种不同的传感器来获取这 18 种不同指标的全部数据，设计了一个硬件传感系统，该系统使用四块 ESP32 微控制器板，集成了 Wi-Fi 和蓝牙连接功能，通过融合来自 14 种不同传感器的各种数据来实现。第三，利用 3D 打印部件将所设计的系统开发为可穿戴设备，可安装在臀部和左右脚上。第四，利用基于 Django 和 JavaScript 的 React.js 网络开发框架，在微处理器托管的边缘计算服务器上创建了一个基于网络的数字仪表板，以即时发布数据，并开发了一个图形用户界面（GUI），以提供各种健康相关指标的直观和实时可视化。开发的物联网生物传感系统的准确性通过基准测试和验证，并将从拟议系统获得的结果与从各种商用传感器获得的结果进行比较。验证结果表明，在所考虑的 18 项指标中，拟议系统的大部分指标准确率超过 90%，所有指标的准确率均超过 85%。这项研究是第一项能够将以下 18 项指标实时报告到单一生物传感系统中的研究，为相关知识体系增添了新的内容：心电图 (ECG 或 EKG)、脑电图 (EEG)、脑电图 (EOG)、肌电图 (EMG)、血压计 (PPG)、心率 (HR)、心率变异性 (HRV)、呼吸频率 (RR)、皮肤温度 (ST)、皮肤湿度 (SH)、血糖水平 (BGL)、血压 (BP)、血氧饱和度 (SpO2)、体重压力 (BWP)、身体运动 (BM)、皮肤电活动 (EDA)、皮肤电反应 (GSR) 和皮肤传导反应 (SCR)。拟议的系统可提供有关各种生命体征的丰富信息，可广泛用于各种应用，包括监测和评估健康状况；情绪和唤醒状态；精神和认知状态；行为、身体和注意力状态；以及生理状态。所开发的系统并非针对特定行业，而是可用于任何相关行业。本文为可穿戴传感器技术、数据可视化技术和健康监测实践的重大进展奠定了基础。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Sensors and Actuators A-physical 工程技术-工程：电子与电气

CiteScore

8.10

自引率

6.50%

发文量

630

审稿时长

49 days

期刊介绍： Sensors and Actuators A: Physical brings together multidisciplinary interests in one journal entirely devoted to disseminating information on all aspects of research and development of solid-state devices for transducing physical signals. Sensors and Actuators A: Physical regularly publishes original papers, letters to the Editors and from time to time invited review articles within the following device areas: • Fundamentals and Physics, such as: classification of effects, physical effects, measurement theory, modelling of sensors, measurement standards, measurement errors, units and constants, time and frequency measurement. Modeling papers should bring new modeling techniques to the field and be supported by experimental results. • Materials and their Processing, such as: piezoelectric materials, polymers, metal oxides, III-V and II-VI semiconductors, thick and thin films, optical glass fibres, amorphous, polycrystalline and monocrystalline silicon. • Optoelectronic sensors, such as: photovoltaic diodes, photoconductors, photodiodes, phototransistors, positron-sensitive photodetectors, optoisolators, photodiode arrays, charge-coupled devices, light-emitting diodes, injection lasers and liquid-crystal displays. • Mechanical sensors, such as: metallic, thin-film and semiconductor strain gauges, diffused silicon pressure sensors, silicon accelerometers, solid-state displacement transducers, piezo junction devices, piezoelectric field-effect transducers (PiFETs), tunnel-diode strain sensors, surface acoustic wave devices, silicon micromechanical switches, solid-state flow meters and electronic flow controllers. Etc...