Generative artificial intelligence-mediated counselling on first aid for seizures: The performance of publicly available chatbot versus its customised version

Abstract

Background

The potential application of cutting-edge generative artificial intelligence chatbots in the capacity of emergency consultants is gaining growing attention. This study aimed to analyse the quality of advice on first aid for seizures generated by a commercially developed chatbot in comparison with its customised version.

Methods

The baseline version of ChatGPT (model GPT-4o) and the same chatbot customised using a specialised knowledge base and prompt engineering were tested in four scenarios mimicking bystander requests for instructions on how to help a victim with seizures. The scenarios included ongoing seizures and postictal states, with or without consciousness and breathing. A checklist-based evaluation was conducted.

Results

In total, 120 user-to-chatbot dialogues were generated (2 chatbots × 15 dialogues × 4 scenarios). The baseline chatbot always failed to consider the victim’s state, including whether the seizures are continuing, or if the victim in the postictal period is conscious and breathing normally. Its advice was non-selective and inaccurate, with frequent omissions of key recommendations on first aid and suggestions of inadequate measures. The customised chatbot-generated guidance was consistently tailored to the victim’s condition, significantly more precise and completely safe. Depending on the scenario, the mean percentage of chatbot responses that fulfilled the checklist items was 14–49 % for the baseline chatbot and 77–92 % for the customised version (p ≤ 0.039).

Conclusions

Whereas the publicly available version of the chatbot is not acceptable for first aid counselling, its expert-informed customisation ensures high accuracy and safety of generated advice. Further research in this field is advisable.