Automatic prompt design via particle swarm optimization driven LLM for efficient medical information extraction

Abstract

Medical information extraction (IE) is an essential aspect of electronic health records (EHRs), but it is a challenging task that converts plain text into structured knowledge, where domain models struggle to achieve performance. Recently, large language models (LLMs), which have demonstrated remarkable capabilities in text understanding and generation, have emerged as a promising method for handling natural language texts. However, LLMs are too dependent on elaborate prompts, resulting in extensive expert knowledge and manual prompt templates needed. In this work, we propose a novel method for the automatic prompt design, called Particle Swarm Optimization-based Prompt using a Large language model (PSOPL). As an efficient method for medical information extraction from EHRs, PSOPL can allow particle swarm optimization (PSO) to automate design prompts by leveraging LLM’s ability to generate coherent text token-by-token. Specifically, starting with a small number of initial prompts, evolutionary operators in PSOPL guide the LLM to generate new candidate prompts iteratively, and the PSOPL evaluates population fitness to retain the optimal prompts. In this way, PSOPL can achieve prompt evolution without model training and reduce the human effort and requirement for domain knowledge. We conducted experiments for open-source LLMs (e.g., Alpaca-7B, GPT-J-6B) and closed-source LLM (e.g., GLM-4), on public medical datasets (e.g., CMeEE, CMeIE, CHIP-CDEE) covering information extraction tasks (e.g., named Entity recognition, relation extraction, event extraction) to verify the method’s generalizability. The experimental results demonstrate the potential of using PSO-based LLMs to design prompts automatically, allowing for the swift extraction of important information about patients in the EHRs.