Automated Coding of Communication Data Using LLM: Consistency across Subgroups

Abstract

Assessing communication and collaboration at scale depends on a labor-intensive task of coding communication data into categories according to different frameworks. Prior research has established that large language models (LLMs), particularly those from the GPT family, can be directly instructed with coding rubrics to code communication data and achieve accuracy comparable to human raters. However, whether the coding from LLMs perform consistently across different demographic groups, such as gender and race, remains unclear. To address this gap, we introduce three checks for evaluating subgroup consistency of LLM-based coding, and empirically examine them based on data from three types of collaborative tasks. Our results show that LLM-based coding perform consistently in the same way as human raters across gender or racial/ethnic groups, demonstrating the possibility of its use in large-scale assessments of collaboration, communication, and other AI-enabled conversation-based assessments.