ArXiv.org最新文献

筛选

英文中文

Development and Benchmarking of a Blended Human-AI Qualitative Research Assistant 混合人-人工智能定性研究助手的开发与标杆测试

ArXiv.org Pub Date : 2025-10-14 DOI: 10.48550/arxiv.2512.00009

Joseph Matveyenko, James Liu, J.D. Parsons, Brown, Ryan A., Palimaru, Alina, Puri, Prateek

{"title":"Development and Benchmarking of a Blended Human-AI Qualitative Research Assistant","authors":"Joseph Matveyenko, James Liu, J.D. Parsons, Brown, Ryan A., Palimaru, Alina, Puri, Prateek","doi":"10.48550/arxiv.2512.00009","DOIUrl":"https://doi.org/10.48550/arxiv.2512.00009","url":null,"abstract":"Qualitative research emphasizes constructing meaning through iterative engagement with textual data. Traditionally this human-driven process requires navigating coder fatigue and interpretative drift, thus posing challenges when scaling analysis to larger, more complex datasets. Computational approaches to augment qualitative research have been met with skepticism, partly due to their inability to replicate the nuance, context-awareness, and sophistication of human analysis. Large language models, however, present new opportunities to automate aspects of qualitative analysis while upholding rigor and research quality in important ways. To assess their benefits and limitations - and build trust among qualitative researchers - these approaches must be rigorously benchmarked against human-generated datasets. In this work, we benchmark Muse, an interactive, AI-powered qualitative research system that allows researchers to identify themes and annotate datasets, finding an inter-rater reliability between Muse and humans of Cohen's $κ$ = 0.71 for well-specified codes. We also conduct robust error analysis to identify failure mode, guide future improvements, and demonstrate the capacity to correct for human bias.","PeriodicalId":523637,"journal":{"name":"ArXiv.org","volume":"1 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-10-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://arxiv.org/pdf/2512.00009","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147919400","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0