BioLP-bench: Measuring understanding of biological lab protocols by large language models

Language models rapidly become more capable in many domains, including biology. Both AI developers and policy makers are in need of benchmarks that evaluate their proficiency in conducting biological research. However, there are only a handful of such benchmarks, and all of them have their limitations. This paper introduces the Biological Lab Protocol benchmark (BioLP-bench) that evaluates the ability of language models to find and correct mistakes in a diverse set of laboratory protocols commonly used in biological research. To evaluate understanding of the protocols by AI models, we introduced in these protocols numerous mistakes that would still allow them to function correctly. After that we introduced in each protocol a single mistake that would cause it to fail. We then gave these modified protocols to an LLM, prompting it to identify the mistake that would cause it to fail, and measured the accuracy of a model in identifying such mistakes across many test cases. State-of-the-art language models demonstrated poor performance compared to human experts, and in most cases couldn't correctly identify the mistake.