HRI-confusion: A multimodal dataset for modelling and detecting user confusion in situated human-robot interaction

IF 1.4 Q3 MULTIDISCIPLINARY SCIENCES

Data in Brief Pub Date : 2025-09-10 DOI:10.1016/j.dib.2025.112047

Na Li , Jane Courtney , Robert Ross

{"title":"HRI-confusion: A multimodal dataset for modelling and detecting user confusion in situated human-robot interaction","authors":"Na Li , Jane Courtney , Robert Ross","doi":"10.1016/j.dib.2025.112047","DOIUrl":null,"url":null,"abstract":"<div><div>The dataset was collected from 28 participants (17 female, 9 male, and 1 non-binary) for a study aimed at modelling and detecting user social behaviours with different confusion states in task-oriented situated human-robot interaction (HRI). The dataset consists of user facial body video recordings synchronised with user speech across three designed experiment scenarios (Tasks 1 - 3). Each experiment lasted approximately one hour per participant. The videos are segmented into individual clips corresponding to specific experimental conversations under predefined conditions: general confusion and non-confusion for Task 1 and 3; and productive confusion, unproductive confusion, and non-confusion for Task 2.</div><div>In total, the dataset contains 789 video clips (body: 392, face: 397). Each video is recorded in high-definition RGB format, capturing user facial expressions or body language along with their speech. These multimodal data provide a valuable resource for studying user cognitive and mental states in human-robot interaction and human-computer interaction.</div><div>The data collected for Task 2 was used in [9]. In compliance with GDPR (General Data Protection Regulation) and DPIA (data protection impact assessment) guidelines, the dataset is freely available upon request at <span><span>https://sites.google.com/view/hridatarequst/home</span><svg><path></path></svg></span>.</div></div>","PeriodicalId":10973,"journal":{"name":"Data in Brief","volume":"62 ","pages":"Article 112047"},"PeriodicalIF":1.4000,"publicationDate":"2025-09-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Data in Brief","FirstCategoryId":"1085","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2352340925007693","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"MULTIDISCIPLINARY SCIENCES","Score":null,"Total":0}

引用次数: 0

Abstract

The dataset was collected from 28 participants (17 female, 9 male, and 1 non-binary) for a study aimed at modelling and detecting user social behaviours with different confusion states in task-oriented situated human-robot interaction (HRI). The dataset consists of user facial body video recordings synchronised with user speech across three designed experiment scenarios (Tasks 1 - 3). Each experiment lasted approximately one hour per participant. The videos are segmented into individual clips corresponding to specific experimental conversations under predefined conditions: general confusion and non-confusion for Task 1 and 3; and productive confusion, unproductive confusion, and non-confusion for Task 2.

In total, the dataset contains 789 video clips (body: 392, face: 397). Each video is recorded in high-definition RGB format, capturing user facial expressions or body language along with their speech. These multimodal data provide a valuable resource for studying user cognitive and mental states in human-robot interaction and human-computer interaction.

The data collected for Task 2 was used in [9]. In compliance with GDPR (General Data Protection Regulation) and DPIA (data protection impact assessment) guidelines, the dataset is freely available upon request at https://sites.google.com/view/hridatarequst/home.

查看原文本刊更多论文

hri混淆：一个多模态数据集，用于建模和检测位于人机交互中的用户混淆

数据集来自28名参与者（17名女性，9名男性，1名非二元），旨在建模和检测任务导向情境人机交互（HRI）中不同困惑状态的用户社会行为。该数据集包括用户面部身体视频记录，并在三个设计的实验场景（任务1 - 3）中与用户语音同步。每个实验持续大约一个小时。视频被分割成单独的片段，对应于预定义条件下的特定实验对话：任务1和任务3的一般混淆和非混淆；以及有效混淆，非有效混淆和非混淆。该数据集总共包含789个视频片段（主体：392，面部：397）。每个视频都以高清RGB格式录制，捕捉用户的面部表情或肢体语言以及他们的讲话。这些多模态数据为研究人机交互和人机交互中的用户认知和心理状态提供了宝贵的资源。为Task 2收集的数据在[9]中使用。根据GDPR（通用数据保护条例）和DPIA（数据保护影响评估）指南，该数据集可在https://sites.google.com/view/hridatarequst/home上免费获取。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Data in Brief MULTIDISCIPLINARY SCIENCES-

CiteScore

3.10

自引率

0.00%

发文量

996

审稿时长

70 days

期刊介绍： Data in Brief provides a way for researchers to easily share and reuse each other''s datasets by publishing data articles that: -Thoroughly describe your data, facilitating reproducibility. -Make your data, which is often buried in supplementary material, easier to find. -Increase traffic towards associated research articles and data, leading to more citations. -Open up doors for new collaborations. Because you never know what data will be useful to someone else, Data in Brief welcomes submissions that describe data from all research areas.