Development of an Assessment Scale for Measurement of Usability and User Experience Characteristics of Bing Chat Conversational AI

IF 2.8 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Future Internet Pub Date : 2023-12-23 DOI:10.3390/fi16010004

G. Bubaš, Antonela Čižmešija, Andreja Kovačić

{"title":"Development of an Assessment Scale for Measurement of Usability and User Experience Characteristics of Bing Chat Conversational AI","authors":"G. Bubaš, Antonela Čižmešija, Andreja Kovačić","doi":"10.3390/fi16010004","DOIUrl":null,"url":null,"abstract":"After the introduction of the ChatGPT conversational artificial intelligence (CAI) tool in November 2022, there has been a rapidly growing interest in the use of such tools in higher education. While the educational uses of some other information technology (IT) tools (including collaboration and communication tools, learning management systems, chatbots, and videoconferencing tools) have been frequently evaluated regarding technology acceptance and usability attributes of those technologies, similar evaluations of CAI tools and services like ChatGPT, Bing Chat, and Bard have only recently started to appear in the scholarly literature. In our study, we present a newly developed set of assessment scales that are related to the usability and user experiences of CAI tools when used by university students, as well as the results of evaluation of these assessment scales specifically regarding the CAI Bing Chat tool (i.e., Microsoft Copilot). The following scales were developed and evaluated using a convenience sample (N = 126) of higher education students: Perceived Usefulness, General Usability, Learnability, System Reliability, Visual Design and Navigation, Information Quality, Information Display, Cognitive Involvement, Design Appeal, Trust, Personification, Risk Perception, and Intention to Use. For most of the aforementioned scales, internal consistency (Cronbach alpha) was in the range from satisfactory to good, which implies their potential usefulness for further studies of related attributes of CAI tools. A stepwise linear regression revealed that the most influential predictors of Intention to Use Bing Chat (or ChatGPT) in the future were the usability variable Perceived Usefulness and two user experience variables—Trust and Design Appeal. Also, our study revealed that students’ perceptions of various specific usability and user experience characteristics of Bing Chat were predominantly positive. The evaluated assessment scales could be beneficial in further research that would include other CAI tools like ChatGPT/GPT-4 and Bard.","PeriodicalId":37982,"journal":{"name":"Future Internet","volume":"21 6","pages":""},"PeriodicalIF":2.8000,"publicationDate":"2023-12-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Future Internet","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.3390/fi16010004","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}

引用次数: 0

Abstract

After the introduction of the ChatGPT conversational artificial intelligence (CAI) tool in November 2022, there has been a rapidly growing interest in the use of such tools in higher education. While the educational uses of some other information technology (IT) tools (including collaboration and communication tools, learning management systems, chatbots, and videoconferencing tools) have been frequently evaluated regarding technology acceptance and usability attributes of those technologies, similar evaluations of CAI tools and services like ChatGPT, Bing Chat, and Bard have only recently started to appear in the scholarly literature. In our study, we present a newly developed set of assessment scales that are related to the usability and user experiences of CAI tools when used by university students, as well as the results of evaluation of these assessment scales specifically regarding the CAI Bing Chat tool (i.e., Microsoft Copilot). The following scales were developed and evaluated using a convenience sample (N = 126) of higher education students: Perceived Usefulness, General Usability, Learnability, System Reliability, Visual Design and Navigation, Information Quality, Information Display, Cognitive Involvement, Design Appeal, Trust, Personification, Risk Perception, and Intention to Use. For most of the aforementioned scales, internal consistency (Cronbach alpha) was in the range from satisfactory to good, which implies their potential usefulness for further studies of related attributes of CAI tools. A stepwise linear regression revealed that the most influential predictors of Intention to Use Bing Chat (or ChatGPT) in the future were the usability variable Perceived Usefulness and two user experience variables—Trust and Design Appeal. Also, our study revealed that students’ perceptions of various specific usability and user experience characteristics of Bing Chat were predominantly positive. The evaluated assessment scales could be beneficial in further research that would include other CAI tools like ChatGPT/GPT-4 and Bard.

查看原文本刊更多论文

开发用于测量必应聊天对话式人工智能可用性和用户体验特征的评估量表

自 2022 年 11 月推出 ChatGPT 对话式人工智能（CAI）工具后，人们对在高等教育中使用此类工具的兴趣迅速增长。虽然其他一些信息技术（IT）工具（包括协作和交流工具、学习管理系统、聊天机器人和视频会议工具）的教育用途经常被评估，涉及这些技术的技术接受度和可用性属性，但对 ChatGPT、必应聊天和巴德等 CAI 工具和服务的类似评估最近才开始出现在学术文献中。在我们的研究中，我们介绍了一套新开发的与大学生使用 CAI 工具时的可用性和用户体验相关的评估量表，以及这些评估量表专门针对 CAI 必应聊天工具（即 Microsoft Copilot）的评估结果。以下量表是通过方便抽样（N = 126）的高等教育学生开发和评估的：感知有用性、一般可用性、可学习性、系统可靠性、视觉设计和导航、信息质量、信息显示、认知参与、设计吸引力、信任、人格化、风险感知和使用意向。上述大多数量表的内部一致性（Cronbach alpha）都在令人满意到良好的范围内，这意味着它们对进一步研究 CAI 工具的相关属性具有潜在的实用性。逐步线性回归显示，对未来使用必应聊天（或 ChatGPT）意向最有影响的预测因素是可用性变量 "感知有用性 "和两个用户体验变量--"信任 "和 "设计吸引力"。此外，我们的研究还显示，学生对必应聊天的各种具体可用性和用户体验特征的看法主要是积极的。所评估的评估量表将有助于进一步研究其他 CAI 工具，如 ChatGPT/GPT-4 和 Bard。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Future Internet Computer Science-Computer Networks and Communications

CiteScore

7.10

自引率

5.90%

发文量

303

审稿时长

11 weeks

期刊介绍： Future Internet is a scholarly open access journal which provides an advanced forum for science and research concerned with evolution of Internet technologies and related smart systems for “Net-Living” development. The general reference subject is therefore the evolution towards the future internet ecosystem, which is feeding a continuous, intensive, artificial transformation of the lived environment, for a widespread and significant improvement of well-being in all spheres of human life (private, public, professional). Included topics are: • advanced communications network infrastructures • evolution of internet basic services • internet of things • netted peripheral sensors • industrial internet • centralized and distributed data centers • embedded computing • cloud computing • software defined network functions and network virtualization • cloud-let and fog-computing • big data, open data and analytical tools • cyber-physical systems • network and distributed operating systems • web services • semantic structures and related software tools • artificial and augmented intelligence • augmented reality • system interoperability and flexible service composition • smart mission-critical system architectures • smart terminals and applications • pro-sumer tools for application design and development • cyber security compliance • privacy compliance • reliability compliance • dependability compliance • accountability compliance • trust compliance • technical quality of basic services.