Feasibility and challenges of large language model (LLM)-generated neonatal resuscitation simulations: a multicenter exploratory study.

Abstract

Background: Simulation-based training (SBT) in neonatal resuscitation has positive impact on educational and neonatal outcomes. However, the implementation of well-designed SBT imposes multifaceted demands on instructors. Large language models (LLMs) might have potential in dynamically generating contextual resuscitation scenarios. But gaps exist regarding the feasibility and challenges of LLM-generated simulation scenarios in neonatal resuscitation.

Methods: This is a prospective, multicenter pilot study evaluating the feasibility and challenges of LLM-generated simulation scenarios. Four scenarios including extremely premature infant, placental abruption, born before arrival in hospital and meconium-stained amniotic fluid were generated by ChatGPT-4o and DeepSeek-R1. Four equivalent scenarios were extracted from Neonatal Resuscitation Program® (NRP®) and RETAIN (a serious game platform). Totally these 16 scenarios were written on standardized templates, coded and randomized. Nine independent instructors from 5 centers, who were blinded to the source of the scenarios (ChatGPT, DeepSeek, NRP® and RETAIN), evaluated the performance of scenarios using modified Jeffries Simulation Design Scale (JSDS). AI hallucination and qualitative evaluation were also compared among four groups.

Results: When compared with NRP® scenarios, ChatGPT-generated scenarios demonstrated comparable median scores without statistically significant difference in overall evaluation, whereas DeepSeek and RETAIN had lower scores in overall evaluation, problem-solving efficacy and scenario fidelity. DeepSeek exhibited inferior performance in providing appropriate information. In debriefing design, ChatGPT achieved higher scores than NRP® (effect size r = 0.25, P = 0.02). There was no statistical difference in AI hallucination between the two LLMs. ChatGPT demonstrated strengths in establishing clear objectives and providing structured debriefing frameworks but exhibited deficiencies in the consistent provision of dynamic vital signs. DeepSeek manifested violations in NRP® algorithm.

Conclusions: ChatGPT-generated simulation scenarios might be feasible in facilitating SBT when supervised by NRP® instructors. NRP® deviations and gaps remain in LLM-generated scenarios, which necessitate objective evaluation prior to implementation. Further research assessing educational outcomes and feedback from target learners are essential for the appropriate integration of LLM-generated simulation into SBT.

Clinical trial number: Not applicable.