Text Chunking in Human-AI Teaming of Document Classification for Urban Systems Management Using Large Language Models

Abstract

Urban systems management requires analysing complex textual documentation that necessitates coding and analysis to set requirements and evaluate built environment performance. This study contributes to the research of applying large language models (LLMs) to qualitative coding activities, aiming to reduce resources while maintaining comparable reliability performing like humans. Qualitative coding faces challenges such as resource limitations, bias and consistency between evaluators. We report the application of LLMs to deductively code 10 case documents for 17 common characteristics for the management of urban systems. We utilise whole-text analysis and text-chunk analysis to compare the processing of LLMs with human coding efforts using three OpenAI models. Results indicate that LLMs may perform similarly to human coders when initialised with specific deductive coding contexts. GPT-4o, o1-mini and GPT-4o-mini showed significant agreement with human raters when using a chunking method. The application of GPT-4o and GPT-4o-mini as additional raters with humans showed statistically significant agreement, indicating that the analysis of textual documents benefits from LLMs addition. The novel contribution of this paper is the domain-specific evaluation of a chunk-based prompting approach for deductive coding in urban systems management, validating human–AI teaming performance.