Impact of the large language model and prompt specificity in generating psychiatric multiple-choice questions

Abstract

Introduction

The use of large language models (LLMs) can assist in creating Multiple-Choice Questions (MCQs) for enhancing psychiatric education. This study evaluates the performance of LLMs in generating psychiatry case-based MCQs, focusing on the influence of the model used and the specificity of the prompt.

Material and methods

Two experiments were carried out. In Experiment 1, ChatGPT-3.5 and ChatGPT-4 were used with a general-purpose prompt. In Experiment 2, ChatGPT-4o was used to compare the generic prompt with a psychiatry-specific version. A total of 90 questions were generated in each experiment (45 per condition), balanced across low, moderate, and high difficulty levels. A panel of psychiatry professors assessed the questions for diagnostic accuracy and difficulty.

Results

In Experiment 1, both ChatGPT-3.5 and ChatGPT-4 showed high diagnostic agreement with experts (κ = 0.889 and κ = 0.703), but little correlation with intended difficulty (ρ = 0.104, p = .496; ρ = − 0.087, p = .57). In Experiment 2, the psychiatry-specific prompt with ChatGPT-4o yielded more accurate MCQs (κ = 0.731 and κ = 0.624) and stronger correlations between intended and expert-rated difficulty (ρ = 0.630, p < .001; ρ = 0.436, p < .001). It also improved the inclusion of relevant clinical information and diagnostic diversity.

Conclusion

LLMs can effectively generate psychiatry MCQs, especially when guided by domain-specific prompts. The psychiatry-specific prompt developed in this study is a useful tool for supporting medical education through high-quality, clinically relevant assessments.