Development and evaluation of a lightweight large language model chatbot for medication enquiry.

Abstract

Large Language Models (LLMs) show promise in augmenting digital health applications. However, development and scaling of large models face computational constraints, data security concerns and limitations of internet accessibility in some regions. We developed and tested Med-Pal, a medical domain-specific LLM-chatbot fine-tuned with a fine-grained, expert curated medication-enquiry dataset consisting of 1,100 question and answer pairs. We trained and validated five light-weight, open-source LLMs of smaller parameter size (7 billion or less) on a validation dataset of 231 medication-related enquiries. We introduce SCORE, an LLM-specific evaluation criteria for clinical adjudication of LLM responses, performed by a multidisciplinary expert team. The best performing lighted-weight LLM was chosen as Med-Pal for further engineering with guard-railing against adversarial prompts. Med-Pal outperformed Biomistral and Meerkat, achieving 71.9% high-quality responses in a separate testing dataset. Med-Pal's light-weight architecture, clinical alignment and safety guardrails enable implementation under varied settings, including those with limited digital infrastructure.