Can Large Language Models (LLMs) Act as Virtual Safety Officers?

Abstract

This study examines the reliability of artificial intelligence (AI) systems─specifically, the large language models (LLMs) ChatGPT, Copilot, and Gemini─to provide accurate lab safety advice, a critical need in high-risk environments. We evaluated LLM performance in addressing several chemical safety queries relevant to academic chemistry laboratories across the criteria of accuracy, relevance, clarity, completeness, and engagement. While all the LLMs tested generally delivered clear and accurate guidance, some shortcomings were identified, raising concerns about reliability during safety emergencies or for nonexpert users. Despite these issues, the findings suggest that with further refinement, AI has the potential to become a valuable tool for lab safety that is complementary to a human laboratory safety officer.