Exploring the possibilities and limitations of customized large language model to support and improve cervical cancer screening.

Abstract

Background: The rapid advancement of artificial intelligence, driven by Generative Pre-trained Transformers (GPT), has transformed natural language processing. Prompt engineering plays a key role in guiding model outputs effectively. Our primary objective was to explore the possibilities and limitations of a custom GPT, developed via prompt engineering, as a patient education tool, which delivers publicly available information through a user-friendly design that facilitates more effective access to cervical cancer screening knowledge.

Method: The system was developed using the OpenAI GPT-4 model and Python programming language, with the interface built on Streamlit for cloud-based accessibility and testing. It initially presented questions to testers for preliminary assessment. For cervical cancer-related information, we referenced medical guidelines. Iterative testing optimized the prompts for quality and relevance; techniques like context provision, question chaining, and prompt-based constraints were used. Human-in-the-loop and two independent medical doctor evaluations were employed. Additionally, system performance metrics were measured.

Result: The web application was tested 115 times over a three-week period in 2024, with 87 female (76%) and 28 male (24%) participants. A total of 112 users completed the user experience questionnaire. Statistical analysis showed a significant association between age and perceived personalization (p = 0.047) and between gender and system customization (p = 0.037). Younger participants reported higher engagement, though not significantly. Females valued guidance on screening schedules and early detection, while males highlighted the usefulness of information regarding HPV vaccination and its role in preventing HPV-related cancers. Independent evaluations by medical doctors demonstrated consistent assessments of the system's responses in terms of accuracy, clarity, and usefulness.

Discussion: While the system demonstrates potential to enhance public health awareness and promote preventive behaviors, encouraging individuals to seek information on cervical cancer screening and HPV vaccination, its conversational capabilities remain constrained by the inherent limitations of current language model technology.

Conclusions: Although custom GPTs can not substitute a healthcare consultations, these tools can streamline workflows, expedite information access, and support personalized care. Further research should focus on conducting well-designed randomized controlled trials to establish definitive conclusions regarding its impact and reliability.

Clinical trial number: Not applicable.