Harnessing Generative AI for Assessment Item Development: Comparing AI-Generated and Human-Authored Items

Abstract

The use of generative AI, specifically large language models (LLMs), in test development presents an innovative approach to efficiently creating technical, knowledge-based assessment items. This study evaluates the efficacy of AI-generated items compared to human-authored counterparts within the context of employee selection testing, focusing on data science knowledge areas. Through a paired comparison approach, subject matter experts (SMEs) were asked to evaluate items produced by both LLMs and human item writers. Findings revealed a significant preference for LLM-generated items, particularly in specific knowledge domains such as Statistical Foundations and Scientific Data Analysis. However, despite the promise of generative AI in accelerating item development, human review remains critical. Issues such as multiple correct answers or ineffective distractors in AI-generated items necessitate thorough SME review and revision to ensure quality and validity. The study highlights the potential of integrating AI with human expertise to enhance the efficiency of item generation while maintaining psychometric standards in high-stakes environments. The implications for psychometric practice and the necessity of domain-specific validation are discussed, offering a framework for future research and application of AI in test development.